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EXECUTIVE  SUMMARY 


This  report  is  one  of  a  series  describing  a  program  for  the  development  and 
validation  bf  a  methodology  for  estimation  and  evaluation  of  operator  workload  (OWL)  in 
Army  systems.  It  presents  the  results  of  Task  2  of  Analytics'  contract  with  the  Army 
Research  Institute  (ARI)  to  "Identify  Army  Requirements  Regarding  OWL,  Select  Specific 
Army  Systems  to  Analyze  ,  and  Provide  Outline  of  OWL  Products".  Included  are  the 
results  of  component  subtasks:  (2. 1)  Review  Army  Requirements  and  Reports,  (2.2) 
Assess  User  Needs,  (2.3)  Outline  Final  Products,  and  (2.4)  Select  Prototype  Army 
Systems  to  Evaluate.  The  overall  purpose  of  this  report  is  to  characterize  existing  and 
future  Army  requirements  and  needs  for  OWL  assessment,  to  tailor  the  OWL  program  to 
meet  these  requirements  and  needs,  and  to  identify  emerging  Army  systems  that  are 
appropriate  candidates  for  exercising  families  of  OWL  assessment  techniques. 

Based  on  the  review  of  Army  documents  and  regulations,  there  seemed  to  be  a  void 
in  specific  guidance  concerning  the  implementation  of  OWL  assessment  during  the  Materiel 
Acquisition  Process  (MAP).  Such  lack  of  specific  guidance  concerning  OWL  assessment 
was  further  substantiated  in  our  interviews  as  well  as  from  questionnaire  data  with  Army 
personnel  who  play  integral  roles  during  the  MAP.  Our  assessment  of  these  findings  has 
resulted  in  tailoring  the  proposed  products  (e.g..  Outlines  of  OWL  Handbooks  and 
Pamphlet)  to  meet  the  apparent  need  for  OWL  guidance  throughout  the  MAP.  In  addition, 
recommendations  are  offered  in  the  report  for  integrating  our  efforts  with  existing  Army 
programs  (e.g.,  MANPRINT)  to  assure  that  OWL  receives  adequate  consideration 
throughout  the  MAP.  With  respect  to  selecting  prototype  Army  systems  to  evaluate, 
candidate  systems  are  offered  that  allow  a  wide-range  of  OWL  techniques  to  be  employed 
as  well  as  providing  opportunities  to  make  substantial  and  positive  contributions  toward 
impacting  the  design  of  these  systems . 
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1.  INTRODUCTION 


This  report  is  one  of  a  series  describing  a  program  for  the  development  and 
validation  of  a  methodology  for  estimation  and  evaluation  of  operator  workload  (OWL)  in 
Army  systems.  It  presents  the  results  of  Task  2  of  Analytics'  contract  with  the  Army 
Research  Institute  (ARI).  The  components  comprising  this  task  are: 

•  Review  Army  Requirements 

•  Assess  User  Needs 

•  Outline  Final  Products 

•  Select  Prototype  Army  Systems  to  Evaluate 

The  overall  purpose  of  this  report  is  to:  Present  Army  requirements  and  needs  as 
obtained  throughout  document  review  and  interviews  regarding  OWL  issues  and  concerns, 
outlines  of  final  products  based  on  those  requirements  and  needs,  and  to  suggest  emerging 
Army  systems  for  OWL  evaluation. 

1  ■  1  Overview  of  Program  Progress 

The  OWL  program  is  one  of  several  focusing  on  the  practical  problem  of 
determining  what  the  Army  can  and  should  do  to  assure  that  systems  can  be  adequately 
operated  by  prospective  personnel.  As  new  technologies  are  implemented  with  greater 
degrees  of  computer-interaction,  there  is  growing  concern  questioning  the  ability  of 
soldiers  to  operate  these  systems.  With  regard  to  OWL,  pertinent  questions  include,  "How 
much  information  processing,  decision  making  and  other  cognitive  tasks  can  system 
operators  handle?"  and  "Under  what  time  and  other  limitations  may  operators  continue  to 
function  before  overload  and  performance  degradations  occur?"  These  OWL  questions 
need  to  be  addressed  during  system  development  to  avoid  marginal  or  inoperable  systems. 

The  primary  focus  of  the  present  program  is  with  single  operator  workload.  For 
present  purposes,  OWL  may  be  thought  of  as  a  representation  via  predictive  and  empirical 
assessment  techniques  of  a  human  operator's  relative  limitation  in  the  capability  to  perform 
work.  Here,  relative  limitation  implies  a  functional  relation  between  (1)  actual  operator 
performance  in  the  context  of  mission  requirements  and  (2)  the  operator's  ultimate 
performance  capability.  Our  emphasis  is  on  the  cognitive,  perceptual  and  psychomotor 
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aspects  of  workload,  although  it  is  recognized  that  there  are  physical  workload  issues  more 
associated  with  maintainer  tasks  (lifting,  carrying,  etc.). 

A  necessary  step  in  the  process  of  providing  practical  assistance  to  the  Army  is  to 
understand  current  procedures  and  requirements  for  addressing  operator  workload  in  the 
Materiel  Acquisition  Process  (MAP).  This  understanding  will  form  the  basis  for 
development  of  useful  guidance  for  the  Army  community.  To  gain  this  understanding  two 
avenues  were  employed.  First,  the  procedural  aspects  of  the  Army  MAP  were  reviewed  via 
written  documents.  Second,  discussions  were  conducted  with  military  and  civilian 
personnel  within  the  Department  of  the  Army  (DA)  to  further  understanding  of  their  roles 
in  the  MAP,  their  current  methods  of  addressing  OWL,  and  their  thoughts  on  what  could 
be  provided  to  them  that  would  be  most  useful  in  their  jobs.  This  process  of  understanding 
and  its  results  are  discussed  later  in  the  report. 

From  the  comments  and  suggestions  gathered,  draft  outlines  of  the  three  handbooks 
have  been  developed  for  this  report.  The  handbooks  which  will  be  developed  during  a  later 
effort  (Task  5)  will  form  the  basis  of  the  guidance  provided  to  the  Army  users  for 
addressing  operator  workload  The  guidance  will  result  from  a  critical  review  of  the  current 
scientific  literature  concerning  definition,  prediction,  measurement  and  evaluation  of 
workload  which  will  be  accomplished  as  part  of  the  follow-on  in  Task  3.  The  suggested 
guidance  and  methodologies  will  be  validated  subsequently  by  application  to  specific  Army 
systems  (Task  4).  However,  the  systems  have  been  chosen  as  part  of  the  present  effort  so 
that  familiarization  with  system  specifics  and  necessary  coordination  with  the  appropriate 
Army  organizations  can  begin.  As  will  be  seen,  the  task  of  choosing  prospective  systems 
was  assisted  by  the  interview  process;  those  most  familiar  with  what  systems  are  most 
appropriate  were  some  of  the  same  individuals  with  whom  we  were  speaking  about  OWL 
requirements  and  needs.  The  application  of  the  techniques  to  the  three  selected  systems 
will  serve  as  the  means  to  validate  the  practical  approach  being  devised  as  well  as  providing 
benefits  for  the  systems  and  the  Army. 

1.2  Organization  of  Report 

The  body  of  this  report  presents  the  results  of  Task  2.  Section  2,  in  particular, 
discusses  the  review  of  documents  that  describe  the  Army  Materiel  Acquisition  Process 
and  how  operator  workload  is  currently  addressed  within  the  MAP.  Subsequently,  the 
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results  of  the  interviews  of  members  of  the  Army  community  concerning  their  roles  and 
responsibilities  (both  current  and  projected)  in  the  MAP  and  their  viewpoints  concerning 
OWL  are  presented  in  Section  3.  In  Section  4,  the  concepts  and  rationale  for  the  final 
products  are  discussed  and  the  product  outlines  are  presented.  The  descriptions  of  Army 
systems  that  have  been  selected  for  assessment  and  validation  of  OWL  assessment 
techniques  are  presented  in  Section  5.  Finally,  Sections  6  and  7  discuss  other  progress  to 
date  as  well  as  plans  for  future  tasks  of  the  OWL  program. 
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2.  REVIEW  OF  ARMY  REQUIREMENTS 


2.1  Introduction 


The  Army  maintains  a  continuous  effort  to  field  the  most  capable  force  possible. 
When  changes  in  doctrine,  organization  or  training  cannot  eliminate  identified  deficiencies, 
then  a  decision  may  be  made  to  eliminate  the  deficiencies  through  equipment  acquisition 
(i.e.,  procuring  hardware  systems).  The  process  by  which  new  equipment  is  conceived, 
designed,  developed  and  procured  is  given  in  regulations,  directives,  and  other 
documents. 

As  part  of  the  OWL  project  effort,  a  review  of  the  documents  and  regulations  that 
drive  Army  materiel  acquisition  was  performed.  The  review  was  one  method  to  gain  an 
understanding  of  the  Army  system  of  materiel  acquisition  --  the  way  the  requirements  for 
materiel  are  developed,  the  information  needed  to  develop  requirements,  how  the 
requirements  are  translated  to  hardware  design  and  development,  and  the  roles  of  members 
of  the  Army  community  who  manage  or  perform  these  tasks.  The  document  review  also 
was  a  means  of  identifying  guidance  inconsistencies  and  voids. 

This  chapter  describes  the  approach  used  to  understand  the  Army  acquisition 
process,  how  the  issue  of  OWL  is  currently  addressed  in  the  process,  what  documents 
provide  guidance  and  where  we  see  OWL  issues  potentially  being  addressed  in  the 
acquisition  process. 


2.2  Approach 

A  series  of  documents  were  reviewed  for  two  major  purposes.  First,  we  needed  to 
assure  a  comprehensive  understanding  of  the  documented  Army  Materiel  Acquisition 
Process  (MAP).  For  this  purpose,  a  detailed  review  of  both  directly  and  indirectly  relevant 
documents  was  conducted  to  identify  (1)  if  and  where  OWL  issues  are  considered  and, 
(2)  how  they  are  currently  addressed. 

The  process  was  an  iterative  one  --  as  more  relevant  documents  were  identified, 
ones  read  previously  were  reread  for  increased  information  and  understanding. 
Additionally,  the  interview  and  questionnaire  methodology  employed  to  assess  Army  user 


f - T. - \ 


2-1 


needs  (see  Section  3.)  yielded  information  that  added  to  our  understanding  of  both  the 
MAP  and  associated  issues  regarding  the  assessment  of  OWL.  The  interviews  provided 
for  iterative  document  review  as  well  as  further  contacts. 

The  following  sections,  then,  will  present  discussion  of  our  review.  The 
discussion  of  the  Army  Materiel  Acquisition  Process  in  Section  2.3  deals  with  the  process 
as  described  in  Army  Regulations  (ARs)  and  other  written  documents.  Of  particular 
importance  is  the  recent  emphasis  on  the  Army  Streamlined  Acquisition  Process  (ASAP) 
and  other  acquisition  alternatives,  such  as  Non-developmental  Items  (NDI)  and  product 
improvements  (P3I,  PIP).  Section  2.4  describes  the  few  places  in  the  reviewed  documents 
where  OWL  is  mentioned.  In  addition,  the  review  was  expanded  to  include  mention  of 
Human  Factors  Engineering  (HFE)  and  MANPRINT  where  OWL  concerns  may  be 
expected  to  be  of  greatest  interest  and  importance.  Finally,  several  conclusions  are  drawn 
from  the  document  review  of  OWL  and  recommendations  are  made. 

Some  documents  are  more  important  than  others  and  specific  reference  will  identify 
those  when  appropriate.  In  all  cases,  the  documents  reviewed  were  the  latest  editions  that 
could  be  obtained.  It  is  noteworthy,  as  can  be  seen  by  the  effective  dates  of  many  of  the 
ARs,  much  guidance  has  been  recently  revised  and  published.  In  one  sense  then,  this 
discussion  is  based  on  guidance  available  within  the  present  slice  of  time.  However,  the 
general  process  of  identifying  requirements  and  developing/acquiring  equipment  to  fulfill 
the  requirements  remains  essentially  unchanged.  The  discussions  that  follow  will  make 
specific  reference  to  the  latest  guidance  available,  but  will  also  include  an  overall 
perspective  of  the  Army  MAP  and  OWL  issues  in  MAP. 

2.3  The  Army  Materiel  Acquisition  Process 


2.3.1  Introduction 


There  has  been  a  great  deal  of  innovation  and  adjustment  in  the  acquisition  process 
during  this  decade.  Incorporation  of  the  recommendations  of  bodies  such  as  the  Packard 
Commission  have  resulted  in  many  changes  in  the  way  the  Department  of  Defense  and  the 
Army  approach  developing  and  fielding  new  materiel.  Four  broad  categories  of  acquisition 
methods  are  recognized.  These  are: 
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•  Traditional  Process 

•  Army  Streamlined  Acquisition  Process  (ASAP) 

•  Non-Development  Items  (NDI) 

•  Product  Improvement. 

The  traditional  process  is  described  in  a  degree  of  detail  which  may  seem  out  of 
proportion  to  its  modem  application.  This  approach  is  taken  because  most  persons  with 
some  experience  in  materiel  acquisition  are  familiar  with  the  traditional  process  and  use  it 
for  comparisons.  The  other  development  methods  are  described  using  the  traditional 
process  as  a  baseline.  This  section  describes  the  Army  Materiel  Acquisition  Process 
(MAP)  and  projects  how  OWL  considerations  could  or  should  be  integrated  into  the  MAP. 

Major  reference  sources  for  this  section  are  AR  70-1,  Systems  Acquisition  Policy 
and  Procedures.  AR  71-9,  Materiel  Objectives  and  Requirements  and  DARCOM  (now 
AMC)  /  TRADOC  Pamphlet  70-2,  Materiel  Acquisition  Handbook.  Major  implementing 
regulations  in  specific  disciplines,  such  as  AR  70-10,  Test  and  Evaluation,  and  AR  602-2 
MANPRINT  have  also  been  useful  sources.  AMC  Regulation  70-52,  System 
Engineering,  is  a  relatively  new  publication  which  may  have  impact  on  incorporating  OWL 
enhancements  to  new  or  improved  materiel.  It  provides  renewed  emphasis  to  the 
importance  of  the  systems  engineering  process  in  materiel  development.  Department  of  the 
Army  Pamphlet  11-25,  Life  Cycle  System  Management  Model  for  Army  Systems,  has 
also  been  useful.  This  latter  reference  is,  howeyer,  over  ten  years  old  and  must  be  used 
with  great  caution.  Many  of  the  features  of  the  model  have  been  significantly  impacted  by 
recent  changes  in  the  acquisition  process.  For  instance,  the  MANPRINT  program  has  had 
a  considerable  amount  of  influence  on  the  treatment  of  manpower,  training,  human  factors 
and  safety  issues.'  Some  caution  must  also  be  used  in  applying  DARCOM/TRADOC 
Pamphlet  70-2. 

2.3.2  The  Traditional  Materiel  Acquisition  Process 

2.3. 2.1  Overview  (AR  70-1. Pam  70-2) 

The  Traditional  acquisition  process  is  divided  into  five  phases.  They  are  the  (1) 
Program  Initiation,  (2)  Concept  Exploration,  (3)  Demonstration  and  Validation,  (4)  Full 
Scale  Development  (FSD),  and  (5)  Production  and  Deployment  Phases.  Production  and 
Deployment  may  be  divided  into  Low  Rate  Initial  Production  and  a  full  Production  and 
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Deployment  Phase.  Each  succeeding  phase  takes  a  materiel  requirement  from  a  broad 
concept  to  successively  more  precisely  specified,  producible  and  supportable  materiel 
solution  to  the  Army's  operational  needs.  The  traditional  process  typically  consumes  about 
1 1  to  15  years  from  program  initiation  to  fielding. 

OWL  considerations  and  trade-offs  should  be  made  early  in  this  life  cycle.  A 
commonly  accepted  rule  of  thumb  holds  that  70  percent  of  system  life  cycle  costs  are  set  at 
or  before  the  system  proceeds  into  the  Demonstration  and  Validation  Phase  (  Advanced 
Development ).  Paralleling  this  is  the  observation  of  a  similar  accelerating  decline  in  design 
flexibility.  Figure  2.3.2- 1  illustrates  the  traditional  process  and  where  OWL  evaluations 
and  predictions  may  be  used  most  appropriately.  OWL  consideration  must  be  made  in  the 
early  phases  of  development  because  the  costs  of  modifying  design  later  accelerate  and 
options  similarly  decline  as  the  system  proceeds  further  into  the  MAP. 

2.3.2.2  Program  Initiation  CAR  70-1.  AR  71-9.  PAM  70-21 

Activities  which  result  in  program  initiation  activities  can  be  divided  into  two  broad 
categories.  These  are  mission  and  operations  oriented  activities  and  science  and  technology 
oriented  activities.  Mission  and  operations  oriented  activities  include  threat  assessments, 
conduct  of  Mission  Area  Analyses  (MAA),  development  of  long  range  plans  such  as  the 
Battlefield  Development  Plan  (BDP)  and  the  Mission  Area  Materiel  Plan  (MAMP),  and 
preparation  of  the  DA  Long  Range  Research  Development  and  Acquisition  Plan 
(LDRRDP). 


OWL  IN  THE  NORMAL  MATERIEL  LIFE  CYCLE 


igure  2.3.2- 1  OWL  Considerations  in  the  Traditional  MAP 


Science  and  technology  oriented  activities  include  basic  and  exploratory  research 
and  other  technical  base  activities  which  lead  to  new  or  improved  technological  capabilities. 
The  operations  and  mission  oriented  activities  may  lead  to  the  development  and  submission 
of  a  Justification  for  Major  Systems  New  Start  (JMSNS),  upon  which  a  major  DoD 
program  may  be  based.  An  Operational  and  Organizational  Plan  (O&O  Plan),  is  the  basis 
for  initiating  a  designated  acquisition  program  (DAP)  at  the  Department  of  Army  level  or 
non-major  development  program  at  levels  below  DA.  The  O&O  Plan  is  also  a  part  of  the 
JMSNS.  OWL  considerations  have  potential  contributions  to  all  these  activities.  For 
example,  basic  and  exploratory  research,  and  technical  base  programs  may  directly  address 
OWL  issues.  Mission  Area  Analyses  (MAAs)  may  conclude  that  operational  work  load 
deficiencies  may  constitute  future  program  drivers.  The  O&O  plan  may  be  constrained  by 
OWL  considerations. 

The  mission  oriented  activities  include  the  thirteen  specific  MAAs  which  have  been 
conducted  and  are  periodically  updated  by  the  US  Army  Training  and  Doctrine  Command 
(TRADOC).  These  analyses  provide  an  in-depth  examination  of  the  Army’s  ability  to 
perform  its  fundamental  combat  missions  (e.g.  Close  Combat,  Fire  Support,  Air  Defense, 
NBC,  etc.).  OWL  considerations  may  limit  specific  operational  capabilities  addressed 
within  given  mission  areas.  These  would  be  expected  to  be  statements  of  OWL  limitations 
based  on  evaluation  of  existing  capability. 

The  BDP  and  MAMP  are  broad  examinations  of  the  Army’s  operational  and 
materiel  capabilities.  They  are  based  on  the  MAA  and  prioritize  future  program  efforts 
based  on  the  DA  LRRDAP.  If  a  materiel  solution  is  considered  appropriate  to  address  a 
mission  area  deficiency,  that  solution  is  supported  in  an  O&O  Plan.  The  O&O  plan  may 
be  prepared  as  a  stand-alone  document  or  may  be  an  attachment  to  the  JMSNS.  OWL 
considerations  play  an  important  role  in  developing  the  O&O  plan.  The  O&O  plan 
(Appendix  C,  AR  71-9)  describes  how  a  system  will  be  integrated  into  the  force  stmcture, 
and  deployed,  operated  and  supported  during  both  peace  time  and  war  time.  It  ultimately 
supports  the  preparation  of  detailed  integrated  logistic  support  planning,  basis  of  issue 
planning,  and  broad  personnel  planning.  MANPRINT  considerations  and  personnel 
impacts  are  a  specific  section  within  the  O&O  Plan.  Assessment  of  these  personnel  impacts 
need  to  be  supported  by  OWL  predictions  and  evaluations  for  similar  existing  systems. 

Technical  base  activities  include  basic  research  and  exploratory  development  which 
address  questions  which  impact  on  future  operational  capabilities.  OWL  issues  need  to  be 
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addressed  within  the  framework  of  these  research  programs.  Technical  base  activities  are 
prioritized  and  funded  in  accordance  with  Science  and  Technology  Objectives  (STOs) 
which  are  established  and  prioritized  in  the  HQ  DA  LRRDAP.  The  STOs  are  based  on 
operational  deficiencies  noted  in  MAAs. 

Program  initiation  results  when  it  is  recognized  that  an  operational  deficiency  exists 
and  it  is  likely  that  a  materiel  solution  will  be  effective.  Typically,  a  Special  Task  Force 
(STF),  Special  Study  Group  (SSG)  or  acquisition  team  is  formed  on  approval  of  the  O&O 
Plan  or  JMSNS.  Secretary  of  Defense  guidance  may  be  established  for  major  programs  in 
a  Program  Decision  Memorandum  (PDM).  The  PDM  provides  broad  program  guidance 
from  the  DoD  level  for  systems  for  which  a  JMSNS  is  required.  The  STF/SSG  or 
Acquisition  Team  carry  the  program  responsibilities  until  the  program  enters  the  Concept 
Exploration  Phase. 

2.3.23  Concept  Exploration  CAR  70-1.  AR  71-9.  AMC  Reg  70-52.  Pam  70-2, 
AR  602-21 

During  concept  exploration,  technical  alternatives  to  meeting  the  stated  material  need 
and  supporting  that  system  are  identified  and  explored.  The  competing  alternatives  include 
incorporation  of  new  technology,  adoption  of  non-development  items  (NDI),  and  product 
improvement  of  existing  hardware.  Market  surveys  are  conducted,  bread  board  and 
experimental  prototypes  are  developed  and  tested,  and  information  regarding  development 
risks  and  program  alternatives  are  developed  and  assessed.  Final  system  concepts  are 
ultimately  selected  and  plans  addressing  training,  logistics,  and  future  testing  are 
developed.  An  Acquisition  Strategy  (AS)  is  developed  which  guides  the  conduct  of  the 
future  program. 

The  concept  exploration  phase  may  be  the  most  critical  for  the  application  of  OWL 
concepts.  Trade  offs  addressing  technical  approaches  and  training  and  support  approaches 
will  impact  on  virtually  all  future  related  development  activities.  OWL  issues  addressed  in 
the  trade  off  determination  (TOD),  trade  off  analysis  (TOA)  and  development  of  the  best 
technical  approach  (BTA)  will  become  embedded  in  the  program  for  its  entire  life.  OWL 
considerations  will  also  impact  on  other  studies  and  analysis  conducted  during  this  phase. 
System  engineering  activities  commence  and  serve  as  a  significant  tool  for  integrating  OWL 
requirements  into  the  system.  Update  of  the  O&O  plan  and  development  of  cost  and 
operational  effectiveness  analysis,  development  of  base  line  cost  estimates  and  independent 
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parametric  cost  estimates  will  all  be  impacted  by  OWL  considerations.  Typical  concept 
formulation  activities  are  described  briefly  with  comments  on  their  relationship  to  operator 
workload  issues.  These  activities  include  : 

•  O&O  Plan  —  The  previously  developed  O&O  Plan  is  revised. 
Requirements  stated  within  the  O&O  Plan  should  be  based  on  an 
understanding  of  potential  OWT,  impacts.  OWL  assessments  based  on 
similar  systems  may  be  employed  in  developing  the  plan. 

•  Concept  Formulation  Package  —  The  CFP  consists  of  the  Trade  Off 
Determination  (TOD),  the  Trade  Off  Analysis  (TO A),  the  Best  Techmcal 
Approach  (BTA),  and  the  Cost  and  Operational  Effectiveness  Analysis 
(COEA).  These  analyses  are  packaged  as  a  part  of  the  CFP  under  a 
cover  letter  which  summarizes  essential  system  features  to  include 
MANPRINT  requirements  (Appendix  E,  AR  71-9),  and  performance 
characteristics. 

•  Trade  Off  Determination  --  The  TOD  establishes  viable  trade  offs  for  the 
suggested  approach  in  pursuing  the  development  program.  It  includes 
life  cycle  costing  and  scheduling  information.  ILS  and  MANPRINT 
requirements  are  included  in  issues  which  must  be  addressed  within  the 
TOD.  OWL  predictions  regarding  one  or  more  trade  off  alternatives 
may  be  required. 

•  Trade  Off  Analysis  -  TOA  is  prepared  jointly  by  the  combat  developer 
and  material  developer.  TOA  serves  to  select  the  best  technical  approach 
based  on  the  alternatives  presented  in  the  trade  off  determination. 

Again,  MANPRINT  and  ILS  requirements  are  essential  trade  offs. 

OWL  predictions  which  consider  system  impacts  resulting  from  OWTL 
considerations  may  be  important  in  completing  the  TOA. 

•  Best  Technical  Approach  —  BTA  is  prepared  jointly  by  the  materiel  and 
combat  developer.  It  documents  the  BTA  to  include  ILS  and 
MANPRINT  concepts  based  on  the  results  of  the  TOD  and  TOA.  The 
results  of  previous  OWL  predictions  are  considered  in  formalizing  the 
BTA. 

•  Cost  and  Operational  Effectiveness  Analysis  —  The  COEA  is  prepared 
by  the  combat  developer.  It  examines  the  cost  and  operational 
effectiveness  of  competing  alternatives.  All  important  systems  aspects 
should  be  considered  in  addressing  COEA  to  include  OWL.  OWL 
predictions  and  the  results  of  OWL  evaluations  on  bread  board  and 
brass  board  prototypes  can  provide  important  information. 

•  Bread  Board/Brass  Board  Tests  --  Bread  board  and  brass  board 
prototypes  are  normally  fabricated  and  tested  during  the  course  of 
concept  evaluation  .  Early  OWL  evaluations  against  these  prototypes 
will  provide  an  important  baseline  for  future  OWL  predictions  and  the 
development  of  future  OWL  evaluation  methodology. 

•  Acquisition  Strategy  --  The  AS  is  the  basic  program  plan  for  the 
development  program.  It  is  prepared  by  the  material  developer  in 
coordination  with  the  other  members  of  the  acquisition  team.  It 
provides  guidance  on  tailoring  the  acquisition  process  for  the  specific 
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development  and  highlights  potential  risks  and  plans  to  reduce  or 
eliminate  risks.  MANPRINT  is  specifically  addressed  in  the  AS  (AR 
70-1,  paragraph  5-2b).  OWL  is  an  issue  to  be  considered  under 
MANPRINT  analyses. 

•  System  Safety  Program  Plan  -  Safety  and  health  hazard  assessment 
issues  have  been  addressed  and  identified  for  further  resolution  during 
the  development  process.  These  measures  are  addressed  in  the  SSPP. 
Potential  OWL  issues  may  result  from  predictive  analyses.  OWL 
evaluation  may  result  from  bread  board  and  brass  board  prototype  tests. 
Considerations  may  include  operational  constraints,  training 
requirements  and  restrictions. 

•  Integrated  Logistics  Support  -  Integrated  logistics  support  planning  is 
initiated.  This  includes  issues  such  as  maintenance  planning,  manpower 
and  personnel  requirements,  training  operating  personnel,  and 
requirements  for  training  devices.  ILSPs  prepared  during  the  concept 
formulations  phase  may  include  the  results  of  investigations  based  on 
performance  data  from  deployed  systems. 

•  Training  Plans  -  Training  plans  developed  during  the  concept 
exploration  phase  address  alternative  training  concepts.  They  are 
designed  to  highlight  critical  training  areas  for  consideration  during  the 
balance  of  the  development  process,  therefore  contributing  to  the 
ultimate  system  availability,  maintainability,  and  operational  capability. 
Human  factors  considerations  in  general,  and  OWL  considerations  in 
particular  are  important  training  planning  drivers.  OWL  predictions 
based  on  OWL  evaluations  of  early  bread  board  and  brass  board,  and 
similar  systems  currently  in  the  field  are  important  sources  of  data  for 
the  preparation  of  training  plans.  These  plans  include  assessment  of 
appropriate  training  methods,  media,  training  devices,  skill 
qualification,  evaluation  procedures,  and  the  need  for  training 
simulations. 

•  System  MANPRINT  Management  Plan  —  The  SMMP  is  the 
management  document  that  describes  MANPRINT  concerns  and  tasks 
and  analyses  that  need  to  be  conducted  during  the  acquisition  to  ensure 
consideration  of  manpower,  personnel,  training,  human  factors 
engineering,  system  safety,  and  health  hazard  assessment.  These  issues 
are  addressed  and  resolved  or  identified  for  resolution  during  following 
phases.  OWL  assessments  of  existing  hardware  may  be  useful  in 
comparative  analyses.  The  SMMP  is  initiated  and  maintained  by  the 
combat  or  training  developer  and  is  updated  throughout  the  acquisition 
process  (AR  602-2).  (See  Section  2.4.3  for  further  discussion  of  the 
SMMP). 

•  Test  and  Evaluation  Master  Plan  -  Test  and  Evaluation  Master  Plan 
(TEMP)  developed  during  the  concept  formulation  provides  the 
foundation  for  development  and  operational  testing  throughout  the 
balance  of  the  program.  The  TEMP  provides  the  frame  work  for 
showcasing  critical  development  and  operational  issues  which  need  to 
be  addressed  during  future  testing.  Operational  issues  which  may  be 
addressed  during  development  testing,  and  the  expansion  of  the 
development  test  data  base  during  operational  testing  are  important 
issues  for  consideration  within  the  TEMP.  Critical  OWL  issues  (such 
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as  those  identified  in  the  SMMP)  are  highlighted  in  conjunction  with 
other  MANPRINT  and  human  factors  related  test  issues. 

•  System  Engineering  Management  Plan  —  The  System  Engineering 
Management  Plan  (SEMP)  is  prepared  in  accordance  with  AMC 
Regulation  70-52.  The  SEMP  will  provide  a  framework  for  the 
integration  of  system  requirements  including  MANPRINT  and  OWL 
considerations  Application  of  sound  system  engineering  principles, 
based  on  the  SEMP,  will  be  a  required  feature  throughout  Ae  life  of  the 
program. 

*  System  Concept  Paper  -  The  SCP  summarizes  the  activity  of  the 
Concept  Exploration  Phase  and  is  the  basis  for  obtaining  approval  to 
proceed  to  the  next  phase  of  development.  Although  the  SCP  is  very 
brief,  not  exceeding  12  pages,  it  is  a  key  document  in  the  Materiel 
Acquisition  Decision  Process  (MADP).  OWL  issues  would  not 
normally  be  expected  to  be  highlighted  within  the  SCP.  However, 
critical  OWL  issues  may  be  highlighted  under  the  MANPRINT  or  HFE 
paragraphs  in  the  Acquisition  Strategy  (Annex  F). 

Programs  in  the  Concept  Explanation  phase  may  be  under  a  program  manager,  may 
continue  to  be  managed  by  the  SSG/STF,  or  may  be  managed  by  an  acquisition  team 
appointed  by  the  developing  agency.  TRADOC  will  typically  follow  the  program  through 
an  appropriate  TRADOC  Systems  Manager  (TSM)  with  input  from  the  Combat 
Development  and  Training  Development  Directorates  of  the  proponent  school. 

2.3.2.4  Demonstration  and  Validation  Phase  (  AR  70-1.  AR  70-10.  AMC  Reg  70- 
52.  Pam  70-2  ■  AR  602-21 

The  Demonstration  and  Validation  Phase  is  conducted  to  verify  preliminary  designs 
and  engineering,  and  to  accomplish  the  necessary  planning  and  trade  offs  to  reduce  risks  in 
future  development  and  fielding.  ILS  and  MANPRINT  issues  are  addressed  early-on  (AR 
70-1,  paragraph  3-5a).  Emphasis  includes  conducting  trade  offs  among  system 
characteristics,  manpower,  personnel,  training,  and  support  concepts.  Other  important 
tasks  include  preparation  of  the  Required  Operational  Capability  documentation  and 
training  device  requirements.  User  participation  is  important  during  testing  in  order  to 
prove  out  the  O&O  concept.  OWL  concepts  and  considerations  are  applied  throughout  this 
phase  of  development.  It  is  essential  to  thoroughly  apply  OWL  evaluation  capabilities 
during  this  phase  in  order  to  insure  that  OWL  related  trade  offs  are  thoroughly  understood. 
The  cost  of  modifying  designs  for  the  sake  of  OWL  enhancements  becomes  progressively 
more  expensive  as  the  program  proceeds  to  full  scale  development. 
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The  demonstration  and  validation  phase  features  procurement  of  advanced 
development  protot3^es  for  testing  during  development  and  operational  tests  .  Frequently, 
competing  prototypes  will  be  developed,  fabricated,  and  tested.  The  advanced 
development  prototypes  are  designed  based  on  functional  requirements  and  are  consistent 
with  the  Best  Technical  Approach  articulated  in  the  Concept  Formulation  Package. 
Technical  and  operational  tests  provide  an  excellent  forum  for  validating  OWL  predictions 
made  during  the  Concept  Formulation  Phase.  It  is  important  to  ensure  that  OWL  issues  are 
addressed  during  these  tests  in  order  to  develop  a  clear  understanding  of  OWL  issues 
which  need  be  addressed  in  future  development  and  production.  Typical  advanced 
development  activities  are  described  briefly  with  comments  on  their  relationship  to  operator 
workload  issues.  These  activities  include: 

•  Advance  Development  Contract  -  Advanced  development  prototypes 
are  based  on  the  preferred  alternatives  studied  during  concept 
formulation.  Frequently,  two  or  more  competing  prototypes  will  be 
procured  for  development  and  operational  test  and  evaluation.  OWL 
specialists  should  be  involved  in  developing  the  technical  data  package 
for  the  Advanced  Development  Contracts  and  in  evaluating  the  resulting 
proposals.  The  advanced  development  prototypes  provide  an  efficient 
form  for  addressing  the  impacts  and  potential  solutions  to  OWL 
problems  in  fielded  items.  OWL  related  desi^s  features,  OWL  studies, 
and  OWL  evaluations  may  be  all  addressed  within  the  frame  work  of  the 
Advanced  Development  Contracts. 

•  Technical  and  Operational  Tests  --  Technical  and  Operational  tests  are 
conducted  on  advanced  development  prototypes  and  address  critical 
issues  established  in  the  TEMP.  Both  government  and  contractor 
developed  test  data  is  used  to  address  these  issues.  OWL  specialists 
participate  in  or  closely  follow  the  conduct  of  these  tests.  In  those  cases 
where  specific  OWL  issues  are  being  evaluated  within  those  test 
programs,  OWL  specialists  will  develop  test  procedures,  collect  or 
supervise  collection  of  OWL  related  data,  and  report  and  evaluate  the 
results  of  those  tests.  Frequently  the  developer  and  user  address 
respective  critical  issues  during  the  same  test  procedures.  Close 
coordination  with  all  individuals  on  the  test  team  will  enhance  the 
efficient  and  effective  collection  of  OWL  related  data.  Established 
OWL  evaluation  methodology,  or  specific  methodology  developed 
during  previous  development  phases,  will  be  applied  during  technical 
and  operational  tests. 

•  ILSP  Updates  --  The  ILSP,  prepared  during  the  Concept  Formulation 
Phase,  receives  a  complete  update  during  demonstration  and  validation. 

This  update  is  based  on  the  results  of  testing.  The  updated  ILSP 
examines  support  planning  concepts,  establishes  a  baseline  support 
concept,  identifies  support  parameters,  and  examines  potential 
supportability  problems.  Logistic  systems  and  resource  constraints,  and 
recommended  reliability  and  maintainability  parameters,  are  formulated. 
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•  Training  Support  Planning  Update  ~  The  plans  for  training  support  ^e 
updated  by  the  trainer  and  the  material  developer  in  coor(fination  with 
the  combat  developer  and  logistician.  This  update  is  done  in 
conjunction  with  preparation  of  the  tentative  qualitative  and  quantitative 
personnel  requirements  information  (TQQPRI)  which  has  significant 
OWL  impact.  The  plans  for  training  support  will  include  plans  for  new 
equipment  training.  Training  support  plans  establish  the  baseline  for 
future  training  during  full  scale  development,  initial  production  and 
fielding,  and  subsequent  support  of  the  system  in  the  field.  Potential  or 
identified  OWL  problems  may  be  addressed  through  training. 

•  Tentative  Qualitative  and  Quantitative  Personnel  Requirements  Inventoiy 
—  The  TQQPRI  is  prepared  by  the  material  developer.  The  TQQPRI  is 
based  on  human  factors  studies,  logistics  support  analysis,  development 
of  a  system  training  strategy,  and  behavioral  research.  It  describes 
personnel  duties,  tasks  down  to  work  units,  performance  standards,  the 
basis  for  manpower  authorization  factors,  recommended  Military 
Occupational  Specialties  (MOS)  to  include  skill  levels,  and 
recommended  organizations.  The  TQQPRI,  in  conjunction  with  the 
Integrated  Logistics  Support  Plans,  are  essential  in  developing  hardware 
basis  of  issue  plans  (BOIP),  training  device  requirements,  and  other 
training  in  support  issues.  The  TQQPRI  provides  the  most  current 
information  concerning  numbers  and  qualifications  of  personnel 
required  for  employment  support  and  maintenance  of  the  system. 

•  Tentative  Basis  of  Issue  Plan  -  The  TBOIP  is  prepared  by  the  combat 
developer  in  consideration  of  studies  conducted  by  the  combat  developer 
and  the  TQQPRI.  It  serves  as  the  basis  for  future  development  of  how 
the  system  will  be  distributed  and  supported  within  the  Army.  The 
TBOIP  is  developed  on  the  basis  of  the  best  available  information. 

•  Training  Device  Requirement  --  TDR  requirements  are  developed  by  the 
trainer  in  conjunction  with  development  of  training  support  plans  and 
the  TQQPRI.  Training  devices  used  during  testing  and  future  tests  may 
also  be  useful  tools  in  investigating  OWL  issues. 

•  Required  Operational  Capability  —  The  ROC  establishes  essential 
operational,  RAM,  technical,  personnel  and  manpower,  training,  safety 
and  health,  human  factors  engineering,  logistical,  and  cost  information. 
It  is  used  as  a  basis  for  proceeding  with  full-scale  development.  Letter 
Requirements  (LRs)  are  prepared  for  similar  puqjose  for  low  dollar 
value  items.  MANPRINT  issues  are  addressed  in  a  specific  section 
(Paragraph  8)  of  the  ROC.  Operational  trainability  and  the  technical 
feasibility  of  the  proposed  system  is  also  addressed.  OWL  inputs  to  a 
ROC  (or  a  LR)  are  based  on  the  results  of  previous  OWL  ev^uations 
and  current  OWL  predictions.  The  most  appropriate  location  for  OWL 
inputs  would  be  in  the  MANPRINT  section. 

•  Safety  and  Health  Hazard  Assessment  —  The  Safety  and  Health  Haz^d 
Assessment  is  developed  based  on  test  results  and  other  demonstration 
and  validation  phase  activities.  Results  of  OWL  evaluations  conducted 
during  tests  and  potential  OWL  problems  may  be  used  in  updating  the 
Safety  and  Health  Hazard  Assessment. 
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•  Human  Factors  Engineering  Analysis  —  The  HFEA  is  conducted  to 
identify  any  human  factors  problems  associated  with  the  system. 
Suitability  of  the  system  to  proceed  to  the  Full  Scale  Development  Phase 
is  established.  Human  factors  issues,  including  OWL  for  resolution  are 
highlighted. 

•  System  MANPRINT  Management  Plan  -  The  SMMP  is  updated  based 
on  demonstration  and  validation  phase  results.  Plans  for  future 
MANPRINT  related  activity  are  incorporated. 

•  Technical  Data  Package  --  The  technical  data  package  (TDP)  for  a  full- 
scale  engineering  development  is  a  detailed  specification  for  engineering 
development  prototypes.  The  specifications  must  be  in  sufficient  detail 
to  insme  delivery  of  hardware  and  software  which  is  characteristic  of 
that  which  may  be  delivered  from  production  processes.  Such  detailed 
specifications  may  be  useful  in  determining  hardware  design 
characteristies  that  impact  OWL. 

•  Acquisition  Strategy  Update  -  The  AS  developed  during  the  Concept  - 
Formulation  Phase  is  updated  to  support  full-seale  engineering 
development  and  subsequent  production  and  fielding.  MANPRINT 
issues,  examined  during  the  development  of  the  original  AS  are  re¬ 
examined,  updated,  and  expanded.  The  results  of  OWL  evaluations 
conducted  during  the  validation  phase  and  current  OWL  predictions  are 
used  as  a  basis  for  preparing  OWL  related  inputs  to  the  AS. 

•  Test  and  Evaluation  Master  Plan  Update  —  The  TEMP  is  updated  to 
reflect  test  and  evaluation  requirements  for  full-scale  development,  and 
production  and  fielding.  OWL  critical  issues  to  be  addressed  during 
technical  and  operational  testing  and  subsequent  production  related 
testing  must  be  articulated  in  this  TEMP  update.  OWL  critical  issues  are 
based  on  OWL  evaluation  results  from  the  tests  and  predictions  made 
during  the  validation  phase. 

•  Cost  and  Effectiveness  Analysis  —  COEA  is  updated  based  on  the 
results  of  tests  and  studies  conducted  during  the 
demonstration/validation  phase.  The  resolution  of  OWL  issues  may 
have  substantial  impacts  on  COEA  results.  The  results  of  OWL 
evaluations  conducted  during  the  validation  phase  and  current  OWL 
predictions  are  used  as  a  basis  for  the  COEA  update. 

•  System  Engineering  Management  Plan  -  System  Engineering  activities 
continue  in  order  to  insure  integration  of  required  system  features  on  a 
total  system  basis.  The  SEMP  is  updated  to  support  full-scale 
development.  These  activities  are  a  major  tool  for  integrating 
MANPRINT  in  general  and  OWL  considerations  into  the  development 
program. 

•  Decision  Coordinating  Paper  —  The  DCP  summarizes  the  results  of  the 
demonstration/validation  phase,  and  provides  a  recommendation  for 
proceeding  with  development  of  the  system.  It  is  a  key  document  in  the 
MADP  for  obtaining  a  decision  to  proceed  to  FSD.  The  DCP  is  not  to 
exceed  eighteen  pages,  excluding  six  annexes.  The  DCP  includes  a 
description  of  the  alternatives  considered  during  the 
demonstration/validation  phase,  and  a  description  of  the  selected 
alternative  to  include  the  operational  concept  for  the  selected  alternative. 
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Sustainability  and  economy  of  manpower  are  issues  to  be  included  in 
that  discussion.  The  DCP  also  includes  technological  risks  for  the 
selected  alternative  and  how  those  risks  have  been  resolved  in  the 
demonstration/validation  phase.  OWL  inputs  to  the  DCP  are  based  on 
OWL  evaluations  conducted  during  the  demonstration/validation  phase 
and  current  OWL  predictions.  The  DCP  includes  the  AS  (Annex  F)  and 
its  attendant  MANPRINT/Human  Factors  section. 

The  Demonstration  and  Validation  Phase  is  managed  by  a  Project  Manager,  or  by 
an  acquisition  team  appointed  from  within  the  developing  agency.  DoD  major  programs 
and  DAP  will  be  conducted  under  a  DOD  or  DA  chartered  Program  Manager.  TRADOC 
will  typically  continue  to  follow  the  major  systems  through  an  appropriate  TRADOC 
Systems  Manager  (TSM)  with  input  from  the  Combat  Developments  Directorate  of  the 
proponent  school.  IPR  programs  which  are  not  under  project  or  product  managers  will  be 
managed  by  an  acquisition  team  appointed  by  the  developing  agency. 


Full  Scale  Development  provides  an  opportunity  to  completely  evaluate  the  system 
in  the  form  expected  to  be  fielded.  Systems  which  successfully  complete  full  scale 
development  are  type  classified  as  standard  and  are  procured  for  issue  to  the  field.  The 
total  system  is  normally  prototyped,  tested  and  evaluated,  to  include  all  support  systems 
and  software.  These  include  simulators,  training  devices,  computer  equipment,  training, 
and  maintenance  manuals.  OWL  evaluation  expertise  is  required  to  ensure  that  OWL  issues 
have  been  addressed  in  the  full  scale  development  prototypes. 

There  is  limited  opportunity  to  make  changes  in  the  system  which  will  enhance 
OWL  performance  during  this  phase  of  development.  Essential  changes  may  be  considered 
if  there  is  a  significant  OWL  problem.  However,  changes  to  any  part  of  the  system  tend  to 
cascade  throughout  the  system,  to  include  software  and  manuals.  As  a  result,  changes  are 
expensive  and  time  consuming.  Most  changes  not  demonstrated  as  essential  for  meeting 
system  requirements  would  be  considered  during  product  improvement  programs 
conducted  after  initial  production  and  fielding.  OWL  inputs  which  have  an  impact  on 
system  design,  therefore,  are  most  effectively  made  prior  to  entry  into  this  phase.  Typical 
full  scale  development  activities  are  described  briefly  with  comments  on  their  relationship 
to  workload  issues.  These  activities  include: 

♦  Full  Scale  Development  Contract  —  The  full  scale  development  contract 
is  solicited  on  the  basis  of  the  technical  data  package  developed  during 
the  validation  phase.  Full  scale  development  prototypes  are  normally 


typical  in  form  and  function  of  the  expected  production  prototype.  The 
contract  also  calls  for  evaluation  and  delivery  of  a  complete  suite  of 
support  equipment,  to  include  test  equipment,  training  devices, 
software,  manuals,  etc.,  as  well  as  preparation  of  a  technical  data 
package  for  production.  OWL  related  criteria  should  be  one  of  the  bases 
for  ev^uation  of  full  scale  development  proposals. 

Technical  and  Operational  Testing  -  The  complete  system  is  tested  and 
evaluated  against  criteria  documented  in  the  ROC  or  LR  and  developed 
in  the  TEMP.  The  Materiel  Developer,  and  his  contractor  and  technical 
tester  (normally  the  US  Army  Test  and  Evaluation  Command-  TECOM) 
prepare  and  execute  technical  testing  in  accordance  with  the  TEMP. 
The  operational  tester,  either  the  Army  Operational  Test  and  Evaluation 
Agency  (OTEA)  (  for  major  systems)  or  a  designated  Training  and 
Doctrine  Command  (TRADOC)  school,  address  operational/user  issues 
as  specified  in  the  TEMP.  The  resulting  test  reports  are  used  to  prepare 
technical  and  operational  Independent  Evaluations.  Specialists  develop 
tests,  or  make  input  into  more  general  tests,  in  order  to  address  OWL  - 
issues  highlighted  in  the  TEMP.  OWL  criteria  which  serve  as  a  basis 
for  accepting  hardware  or  support  equipment  and  software  may  be 
addressed  in  contractor  testing  for  subsequent  review  by  the 
government.  OWL  evaluation  techniques  are  integrated  as  required  into 
all  testing. 

SMMP  Update  —  The  SMMP  is  updated  throughout  the  cycle  to  reflect 
analyses  performed,  questions  and  concerns  addressed,  new 
MANPRINT  concerns  that  have  been  raised,  as  well  as  maintain  an 
audit  trail  of  all  decisions  and  work  that  has  been  done  in  support  of  the 
SMMP. 

ILSP  Update  -  The  updated  ILSP  is  based  on  the  ILSP  prepared  during 
the  validation  phase.  The  plan  is  expanded  to  address  support  of  the 
system  as  it  is  introduced  into  the  field.  The  ILSP  is  updated  in  concert 
with  updates  to  the  QQPRI,  BOIP,  incorporation  into  tables  of 
organization  and  equipment  (TOE),  preparation  of  the  Materiel  Fielding 
Plan,  and  other  ILS  oriented  actions.  Test  results  and  specific  ILS 
studies  are  also  used  as  a  basis  for  the  update.  The  results  of  OWL 
evaluations  addressed  during  testing  and  other  OWL  studies  provide 
OWL  input. 

TEMP  Update  -  The  TEMP  is  updated  to  support  testing  required 
during  the  production  and  deployment  phase.  Testing  required  to 
demonstrate  that  major  deficiencies  noted  during  technical  and 
operational  tests  are  corrected  is  described.  Production  validation  tests 
and  follow-on  operational  test  and  evaluation  requirements  are 
established.  Requirements  for  first  article  and  initial  production  tests  are 
also  established. 

AS  Update  -  The  Acquisition  Strategy  is  updated  to  emphasize 
production  and  deployment  requirements.  Critical  OWL  issues  which 
remain  to  be  addressed  during  the  production  and  deployment  phase 
may  be  presented  as  a  MANPRINT  consideration. 

Requirement  Documents  Revisions  -  Revisions  to  the  requirements 
document  developed  during  the  Demonstration  and  Validation  Phase 
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(ROC,  LR,  TDR)  must  be  considered  to  support  production  and 
deployment.  Technical  and  operational  characteristics  may  require 
revisions  for  a  number  of  reasons  including  changes  to  the  established 
threat.  Any  changes  to  the  established  requirement  must  be  approved  by 
both  AMC  and  li^DOC  and,  in  conjunction  with  the  MADP,  DA. 

•  Decision  Coordinating  Paper  —  The  DCP  is  prepared  to  support  the 
production  and  deployment  decision.  It  summarizes  the  results  of  full 
scale  development  and  document  recommendations  for  production  and 
deployment.  It  includes  a  full  description  of  the  commitments  the  Army 
is  making  in  proceeding  with  production,  to  include  budget 
requirements  and  future  support  requirements.  TTie  revised  Acquisition 
Strategy,  with  its  required  MANPMNT  and  human  factors  sections,  is 
an  annex  to  the  DCP. 

Typically,  the  same  type  of  program  management  method  is  continued  from  that 
used  in  the  Demonstration  and  Validation  Phase.  Development  programs  may  be  managed 
by  a  program  or  project  manager,  or  by  an  acquisition  management  team  appointed  by  the 
developing  agency.  TRADOC  management  is  also  normally  continued  with  a  TSM 
supported  by  combat  development  and  training  development  elements.  Programs  which 
are  project  managed  normally  continue  with  that  form  of  management,  at  least  until  the 
system  is  successfully  fielded. 

2.3.2.5  Production  and  Deployment  (AR  70-1.  Pam  70-2) 

During  production  and  deployment  operational  quantities  of  the  system  and  required 
support  equipment  is  procured,  personnel  and  operational  units  are  trained,  and  the  logistic 
support  system  is  implemented  to  support  the  system  in  the  field.  If  directed  at  the 
Milestone  III  decision  point,  Low  Rate  Initial  Production  may  be  implemented  to  address 
additional  testing,  production  engineering  or  production  base  issues. 

Follow-on  Operational  Test  and  Evaluation  (FOTE)  may  be  conducted  during  the 
production  and  deployment  phase,  if  required,  to  address  a  spectrum  of  issues,  including 
those  with  manpower  and  training  impacts.  FOTE  has  the  potential  to  serve  as  relatively 
well  controlled  fomm  for  developing  OWL  data  upon  which  to  base  product  improvement 
or  new  equipment  requirement  issues. 
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2.3.3  The  Army  Streamlined  Acquisition  Process  ( AR  70-1 ) 


2.3.3. 1  Overview 

The  Army  Streamlined  Acquisition  Process  (ASAP)  compresses  the  standard 
acquisition  cycle  from  over  eleven  years  to  seven  years  or  less.  The  objective  of  the 
streamlined  process  is  to  achieve  operational  capability  within  the  minimum  practical 
amount  of  time,  for  low  risk  development  programs.  Programs  which  need  more  detailed 
and  deliberate  development  processes,  generally  characterized  as  higher  risk  development 
programs,  may  still  follow  all  or  portions  of  the  "traditional"  acquisition  process. 

The  streamlined  process  is  characterized  by  four  distinct  phases  as  seen  in  Figure 

2.3.3-  1.  They  are  1)  Requirements  and  Technical  Base  Activities,  2)  Proof  of  Principle,  3) 
Development-Production  Prove  Out,  and  4)  Production  and  Deployment.  Proof  of 
Principle  is  followed  by  a  go/no-go  decision  (Milestone  I/II)  to  proceed  into  the 
development/prove  out  phase.  The  focus  of  the  streamlined  process  is  in  the 
development/prove  out  phase.  Activities  typical  of  traditional  full-scale  development  and 
early  production/deployment  are  conducted  as  expeditiously  as  practical. 
Development/prove  out  is  followed  by  a  Milestone  HI  decision  point  in  order  to  provide 
approval  to  enter  full  production  and  deployment.  The  objective  of  the  streamlined  process 
is  to  achieve  a  Milestone  III  decision  in  fours  years  or  less  from  demonstration  of  proof  of 
principle.  Overlaid  on  the  streamlined  acquisition  procedure  are  provisions  for  preplanned 
product  improvements  (P3I)  throughout  the  life  of  the  system.  New  or  improved 
technology  is  "inserted"  at  appropriate  points  throughout  the  life  cycle  as  the  threat  or  the 
technology  changes.  These  insertion  points  may  be  during  the  development/prove  out 
stage,  as  well  as  during  the  production  and  deployment  phase  or  later  in  the  cycle. 

Documentation  prepared  during  the  conduct  of  programs  under  the  streamlined 
acquisition  process  is  similar  to  that  prepared  under  the  traditional  process.  OWL 
considerations  for  analysis  and  the  development  of  design  approaches  and  documentation 
are  identical  in  comparison  to  those  conducted  during  the  traditional  process.  Successive 
iterations  of  documentation,  as  the  system  proceeds  through  the  stages  of  development 
under  the  traditional  process,  are  eliminated.  OWL  considerations  for  the  development  of 
specific  documentation  under  the  streamlined  process  will  not  be  repeated  here.  Figure 

2.3.3-  1  illustrates  how  OWL  considerations  should  enter  the  ASAP;  these  are  identical  to 


OWL  IN  THE  STREAMLINED  LIFE  CYCLE 


Figure  2.3.3- 1  OWL  Considerations  in  the  ASAP 


those  in  the  traditional  process.  It  is  important  to  emphasize  that  as  the  process  is 
compressed,  so  too  are  the  opportunities  for  and  impacts  of  OWL  consideration.  The  most 
important  phase  of  development  under  the  Streamlined  Acquisition  Process  in  considering 
OWL  inputs  is  Proof  of  Principle.  Incorporation  of  OWL  enhancements  will  be  much 
more  cost  effective  and  efficient  during  and  immediately  after  the  proof  of  principle  phase, 
in  comparison  to  later  in  the  Streamlined  Acquisition  Process.  A  brief  description  of  each 
phase  under  the  streamlined  process,  with  emphases  on  OWL  considerations,  is  presented 
below. 


2.3.3.2  Discussion  of  ASAP  Phases 

The  first  phase  is  the  Requirements  Definition/Technical  Base  Activities.  Activities 
conducted  during  this  phase  are  similar  to  the  program  initiation  activities  under  the 
traditional  process.  They  ultimately  result  in  development  of  a  JMSNS,  or,  as  a  minimum, 
an  O&O  Plan,  and  result  in  approval  to  proceed  into  the  proof  of  principle  phase.  Basic 
program  management  documents  such  as  the  Acquisition  Strategy,  Acquisition  Plan, 
SMMP  and  Test  and  Evaluation  Master  Plan  may  also  be  prepared  during  this  phase. 
OWL  considerations  may  drive  requirements,  as  well  as  the  accomplishment  of  technical 
base  activities,  such  as  research  programs  focused  on  OWL  issues.  OWL  predictions,  and 
the  results  of  OWL  evaluations  of  fielded  systems  which  may  address  similar  mission 
requirements,  will  impact  the  TEMP,  SMMP,  O&O  Plan,  and  AS. 

The  ASAP  calls  for  establishing  a  Technology  Integration  Steering  Committee 
(TISC),  with  the  objective  of  comparing  technological  opportunities  with  emerging 
requirements  (AR  70-1,  Paragraph  7-2c(2)).  OWL  considerations  need  to  be  considered 
by  the  TISC.  TISC  activities  ultimately  develop  solutions  which  are  suitable  for 
consideration  in  hardware  under  proof  of  principle.  Additionally,  "Star"  Reviews  provide 
visibility  and  focus  at  the  general  officer  level  at  the  start  of  proof  of  principle.  OWL 
considerations,  based  on  related  technical  base  activities,  are  issues  for  consideration  by 
the  TISC  and  during  Star  Review. 

The  second  phase  is  Proof  of  Principle.  Proof  of  principle  consolidates  activities 
conducted  during  concept  exploration  and  demonstration/validation  under  the  traditional 
process.  The  phase  permits  the  conduct  of  user  demonstrations  and  experimentation 
employing  brassboard  or  surrogate  systems  in  order  to  prove  out  the  technical  approach 
and  operational  concept.  Proof  of  principle  results  in  a  "go/no-go"  decision  to  proceed  into 
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the  development/prove  out  phase.  Documentation  supporting  that  decision  (Milestone  I/II) 
includes  the  CFP,  TEMP,  ILSP,  and  AS.  MANPRINT  and  ILS  issues  are  addressed  on 
the  basis  of  troop  demonstrations  and  experimentation  (AR  70-1,  Paragraph  7-2f(3)). 

Development  of  OWL  requirements,  and  conduct  of  studies  and  analyses  related  to 
establishing  OWL  impacts,  must  receive  strong  emphasis  during  proof  of  principle.  OWL 
evaluations  of  surrogate  systems  and  brassboard  prototypes  serve  as  the  basis  for 
developing  OWL  requirements  as  a  system  enters  the  development/prove  out  phase.  OWL 
predictions,  based  on  systems  currently  in  the  field,  will  also  make  an  important 
contribution  to  understanding  OWL  impacts.  As  with  the  early  phases  of  the  traditional 
MAP,  OWL  considerations  will  have  their  most  cost  effective  impacts  during  proof  of 
principle.  Likewise,  design  features  oriented  to  enhancing  OWL  characteristics  must  be 
identified  during  this  phase.  There  will  be  little  opportunity  for  modifying  designs  in  order 
to  enhance  OWL  characteristics  during  the  development  and  prove  out  phase,  except  under 
the  provision  for  preplanned  product  improvements  (P3I). 

The  third  phase  is  the  Development/Production  Prove  Out  phase.  The 
development/production  prove  out  phase  encompasses  activities  which  are  similar  to  full- 
scale  development  under  the  traditional  process.  System  characteristics  are  demonstrated 
using  hard  tool  prototypes  during  technical  and  operational  testing.  Particular  emphasis  is 
placed  on  ILS,  MANPRINT  (AR  70-1,  Paragraph  1-2%),  and  producibility  engineering  and 
planning.  Documentation,  and  OWL  considerations  to  be  made  during  the  preparation  of 
that  documentation,  is  similar  to  that  required  for  full-scale  development  under  the 
traditional  process.  OWL  predictions  and  the  results  of  OWL  evaluations  are  sources  of 
OWL  data  used  in  preparing  that  documentation.  OWL  evaluation  methodology  is 
employed  during  testing  in  order  to  demonstrate  that  prototypes  meet  required  OWL 
characteristics. 

Throughout  this  phase,  requirements  for  P3I  technology  insertions  are  considered. 
Product  improvements  may  be  made  during  finalization  of  development/production  prove 
out  designs,  or  may  be  delayed  until  further  into  the  production/deployment  phase.  OWL 
enhancements  are  more  likely  to  be  incorporated  in  a  system  which  has  entered  the 
production/deployment  or  the  development/prove  out  phases  as  P3Is.  Based  on  a 
Milestone  III  review  successful  conduct  of  the  development/production  prove  out  phase 
results  in  both  type  classifying  equipment  as  standard,  and  entering  the 
production/deployment  phase. 
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The  final  phase  is  the  Production/Deployment  Phase.  The  production/deployment 
phase  includes  low  rate  initial  production  (LRIP),  first  article  testing  (FAT),  new 
equipment  training,  and  initial  fielding  activities.  Full  rate  production  is  achieved  as  initial 
fielding  is  completed  and  production  models  are  demonstrated  to  achieve  required 
capabilities.  Documentation  required  under  the  streamlined  process  is  similar  to  that 
required  for  the  production/deployment  phase.  Generally,  schedules  for 
production/deployment  under  the  streamlined  process  are  compressed  to  one  and  a  half  to 
two  years.  OWL  evaluation  methodology  may  be  used  to  demonstrate  that  production 
prototypes  meet  required  OWL  characteristics.  OWL  evaluations  of  fielded  hardware  may 
result  in  development  of  requirements  for  product  improvement  as  production/deployment 
is  completed.  These  requirements  normally  would  be  incorporated  as  a  part  of  the  overall 
P3I  program  for  the  system. 

2.3.4  Adoption  of  Non-Developmental  Items  (NDD 

NDI  is  a  candidate  for  fulfilling  any  material  need.  NDI  include  commercially 
available  items,  as  well  as  items  adopted  by  other  services  or  friendly  foreign  nations. 
NDIs  are  considered  in  conjunction  with  market  investigations  accomplished  early  in  the 
development  cycle.  If  pursuing  an  NDI  appears  to  be  an  acquisition  alternative,  a  program 
to  procure,  test,  and  adopt  the  item  is  developed.  There  are  two  general  categories  of  NDI 
(AR  70-1,  Paragraph  7-3d): 

•  Category  A  -  Off  the  shelf  items  which  need  no  further  development  or 
modification  in  order  to  achieve  the  required  operational  capability. 

These  items  would  be  expected  to  be  used  in  a  military  environment 
under  the  same  conditions  for  which  they  were  intended  in  commercial 
environment. 

•  Category  B  —  Off  the  shelf  items  requiring  modification  to  hardware 
designs  or  software  in  order  to  operate  in  the  military  environment. 

These  modifications  are  typically  required  to  ruggedize  the  item  or 
enhance  system  survivability. 

Acquisition  Strategies  for  further  development  and  fielding  of  NDI  consider 
modifications  needed  and  the  requirements  to  demonstrate  and  prove  the  suitability  of  the 
equipment  for  military  use.  Adoption  of  an  NDI  does  not  eliminate  the  need  to  examine 
essential  characteristics  to  include  MANPRINT,  systems  safety,  and  logistic  support 
concepts. 
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Conduct  of  NDI  procurement  programs  are  tailored  to  the  requirement  and  the 
availability  of  suitable  commercial  or  foreign  equipment.  NDI  items  which  require 
considerable  ruggedization  with  other  modifications  may  drive  establishing  a  program  with 
features  and  requirements  similar  to  a  hardware  development  program.  Sufficient  testing 
must  be  conducted  to  prove  operational,  maintenance,  and  support  characteristics.  The 
features  of  the  program  are  developed  on  the  basis  of  the  market  analysis  conducted  prior  to 
Milestone  I. 

ILS  and  MANPRINT  issues  vary  from  procurement  to  procurement.  They  are 
driven  by  the  requirement  and  the  characteristics  of  the  commercial  item  and  the  Army's 
needs.  The  range  of  solutions  varies  from  full  acceptance  of  the  commercial  item,  as  taken 
"from  the  shelf  with  a  full  commitment  to  future  contractor  maintenance  and  training 
support,  to  incorporation  of  the  items  into  the  Army  training  and  logistic  support  system, 
and  all  combinations  in  between.  All  acceptable  commercially  available  data  is  procured 
and  utilized  for  system  support. 

The  NDI  program  is  tailored,  based  on  the  item  characteristics  and  support 
available,  to  insure  all  requirements  are  met.  For  programs  which  need  hardware 
modifications  and/or  development  of  training  and  support  capabilities,  the  AS  may  be  very 
similar  to  ASAP  of  a  full  system.  The  AS  must  address  MANPRINT  and  ILS  issues  and 
resolutions  must  survive  the  MADP  before  adoption  of  an  NDI. 

2.3.5  Product  Improvements  TAR  70-1.  AR  70-15.  Pam  70-21 

Product  improvement  is  a  preferred  method  for  responding  to  materiel 
requirements.  They  may  range  from  modifications  to  a  fielded  item  as  a  result  of  a  Product 
Improvement  Program  (PIP)  to  planned  evolutionary  changes  to  a  system  in  development 
(P3I).  A  PIP  is  distinguished  from  a  P3I  in  that  a  PIP  applies  to  systems  which  are  already 
fielded  and  are  no  longer  in  production.  Product  improvements  are  an  appropriate  method 
of  responding  to  workload  deficiencies  which  are  revealed  late  in  the  development  cycle  or 
in  fielded  systems.  PIPs  and  P3Is  are  prioritized  and  integrated  into  the  overall  Army  RDA 
program  in  the  LRRDAP  (see  paragraph  23.2.2). 

Product  improvements  may  be  applied  to  systems  for  a  variety  of  reasons.  For  a 
PIP,  these  include:  requirements  to  enhance  human  factors,  safety  or  other  MANPRINT 
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related  system  characteristics;  improve  system  technical  characteristics  and  operational 
effectiveness,  as  well  as  expand  weapon  system  effective  life.  Preplanned  Product 
Improvements  (P3I)  are  required  to  be  addressed  in  the  requirements  document  for  all 
developmental  systems.  They  may  be  pursued  to;  reduce  development  time;  reduce 
development  risks;  permit  a  timely  response  to  changing  threats;  as  well  as  respond  to 
emerging  technological  opportunities.  P3Is  are  also  appropriate  during  system 
development  to  incorporate  enhancements  to:  system  performance  characteristics  and 
operational  effectiveness;  safety;  human  factors;  and  other  MANPRINT  requirements. 

PIPs  may  be  initiated  by  the  materiel  developer,  the  combat  developer  or  field 
elements.  The  combat  developer  validates  the  need  which  precipitates  the  product 
improvement  requirement.  A  Product  Improvement  Proposal  is  prepared  and  coordinated 
by  the  materiel  developer.  The  Product  Improvement  Proposal  fully  documents  the  need 
for  improvement  and  plans  for  development  and  application  of  the  modifications.  It 
includes  an  acquisition  strategy  and  plans  for  developer  and  user  testing  to  demonstrate  the 
efficacy  of  the  modifications.  Detailed  PIP  procedures  are  established  in  AR  70-15.  The 
PIP  ultimately  produces  a  DA  Modification  Work  Order  (DAMWO),  Depot  Maintenance 
Work  Request  (DMWR)  or  other  approach  to  applying  the  modification  and  documenting  it 
in  the  system  technical  data  package. 

P3I  encourages  an  evolutionary  approach  to  system  design.  That  evolution  is 
described  in  the  acquisition  strategy  for  the  P3I  program  which  are  pursued  in  three  phases: 

•  Phase  I  establishes  how  the  system  needs  to  evolve  throughout  the  life 
cycle  in  order  to  respond  to  future  operational  requirements  and 
technological  opportunities. 

•  Phase  n  incorporates  the  results  of  Phase  I  into  the  basic  system  design. 

•  Phase  III  applies  the  modifications  as  block  (i.e.,  several  system 
changes)  or  individual  changes. 

Systems  which  have  entered  production  and  fielding  must  also  make  provision  for 
the  modification  of  systems  already  in  the  field. 
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2.4  OWL  Issues  in  the  Acquisirion  Process 


2.4. 1  How  OWL  Issues  are  Currently  Addressed 

One  of  the  main  objectives  of  the  document  review  was  to  determine  if  and  how 
operator  workload  issues  (either  physical  or  mental)  were  currently  addressed  as  part  of 
the  Army  MAP.  Not  surprisingly,  there  is  not  much  specific  discussion  or  guidance  given 
in  the  regulations.  There  are  more  citations  when  the  review  is  expanded  to  include 
mention  and  guidance  concerning  areas  related  to  "workload  issues"  (e.g.,  human  factors 
engineering,  manpower,  personnel,  or  other  topic  or  term  that  considers  soldiers  as  well 
as  hardware).  However,  the  connection  and  implications  regarding  OWL  is  more 
tenuous.  This  section  addresses  the  specific  references  to  workload  that  were  found  in  the 
documents  reviewed.  The  manner  in  which  related  topics  are  addressed  in  documents  is 
also  discussed  in  a  more  general  context  of  OWL. 

Documents  from  DoD,  DA  and  subordinate  organizations  were  identified.  Within  each 
document,  other  documents  are  listed  (as  required  publications)  which  are  necessary  for 
complete  understanding  and  implementation.  In  addition,  there  are  also  reference  or  related 
publications  which  may  contain  useful  information  but  are  not  essential  for  complete 
understanding.  By  comparing  the  lists  of  required  and  related  publications,  a  "document 
tree"  may  be  established  as  graphically  displayed  in  Figure  2.4.1-1.  In  this  "tree",  AR  70- 
1  may  be  seen  to  be  the  key  Army  acquisition  document.  It  is  associated  with  other  ARs 
describing  aspects  of  the  acquisition  process  and  with  DoD  high  level  guidance  for  military 
system  acquisition.  AR  70-1  also  requires  more  technically  oriented  ARs  such  as  AR  602- 
1  and  602-2.  With  regard  to  OWL,  the  most  explicit  reference  to  operator  workload  in 
Army- wide  documents  is  contained  in  MIL-H-46855  (cf.,  Section  2.4. 1.2).  There  are 
specific  Data  Item  Descriptions  associated  with  this  military  specification  and,  in  turn,  this 
military  specification  is  directly  related  to  the  Human  Factors  Engineering  AR.  This 
indicates  that  these  "workload'-related  DIDs  are  only  referenced  via  the  HFE  AR  602-1. 
The  relationship  shown  in  this  document  tree  could  be  kept  in  mind  during  the  more 
thorough  discussion  of  the  ARs,  DoD  guidance,  and  other  related  documents  which 
follows. 
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DoDD  5000.1 
DoDI  5000.2 


^  REFERENCED/REQUIRED  PUBS 

RELATED  PUBS 


Figiire  2.4. 1-1.  OWL-  Related  Relationship  Between  Primary  Documents. 
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2.4. 1 .1  Army  Regulations 


A  review  of  Army  Regulations  (ARs)  was  undertaken  to  identify  those  that  govern 
system  acquisition  and  any  requirements  for  operator  workload  considerations  as  described 
earlier.  The  key  documents  identified  are  listed  in  Table  2.4. 1-1.  Of  these,  the  first  two 
(AR  70-1,  -10)  are  in  the  Research  and  Development  series.  The  next  two  (AR  71-3,  -9) 
are  in  the  Force  Development  series.  ARs  602-1  and  -2  are  in  the  Soldier-Materiel  Systems 
series.  These  regulations  represent  basic  policy,  guidance,  and  required  formats  in  these 
three  areas. 


Table  2.4. 1-1  Key  Army  Regulations 

AR  Title  _ Effective  Date 


AR70-1 

System  Acquisition  Policy  and  Procedures. 

1  Dec  86. 

AR  70-10 

Test  and  Evaluation 

30  Apr  86. 

AR71-3 

User  Testing 

1  Mar  86. 

AR71-9 

Materiel  Objectives  and  Requirements 

20  Mar  87. 

AR  602-1 

Human  Factors  Engineering  Program 

15  Feb  83. 

AR  602-2 

Manpower  and  Personnel  Integration  (MANPRINT) 
in  Materiel  Acquisition  Process 

18  May  87. 

A  brief  summary  of  these  ARs  indicates  the  major  content  and  intention  of  each: 

•  AR  70-1  covers  basic  policies  and  procedures  for  Army  system 
acquisition  and  "emphasizes  front-end  planning  and  tailoring  of  the 
materiel  acquisition  process..."  (p.  3).  The  ASAP  is  introduced  and  its 
policies  and  procedures  described. 

•  AR  70-10  covers  basic  policies  and  procedures  for  test  and  evaluation 
and  provides  information  concerning  test  and  evaluation  for  use  at 
decision  reviews. 

•  AR  71-3  covers  policies  and  assigns  responsibilities  for  user  test  and 
evaluation  and  continuous  comprehensive  evaluation  (C2E)  in  the  MAP. 


•  AR  71-9  covers  policies  and  procedures  and  assigns  responsibilities  for 
preparing  requirements  documents  for  materiel  and  gives  guidance  for 
the  combat  development  process  within  the  MAP. 

•  AR  602- 1  integrates  human  factors  engineering  throughout  the  MAP. 

•  AR  602-2  covers  policies  and  procedures  for  the  M ANPRINT  Program 
and  establishes  the  requirement  for  the  System  MANPRINT 
Management  Plan  (SMMP).  MANPRINT  is  an  umbrella  concept  that 
encompasses  HFE,  manpower,  personnel,  training,  health  hazard 
assessment,  and  system  s^ety. 

These  six  documents  do  not  specifically  address  OWL.  Army  Regulation  70-1 
does,  however,  establish  the  policies,  procedures  and  requirements  for  all  applicable  Army 
system  acquisition  programs.  It  calls  out  both  AR  602-1  and  602-2,  and  specifies 
MANPRINT  inputs  into  the  different  phases  of  the  traditional  Life  Cycle  System 
Management  Model  (LCSMM).  Additionally,  it  specifies  that  MANPRINT  considerations 
must  be  included  in  tailored  acquisition  programs  (e.g.,  ASAP  and  NDI)  and  materiel 
improvements  [i.e..  Engineering  Change  Proposals  (ECP),  Product  Improvement 
Proposals  (PIP),  or  Preplanned  Product  Improvements  (P3I)].  Specifically,  it  provides  that 
"No  acquisition  ...  is  exempt  from  minimal  essential  test  and  evaluation  necessary  to 
verify  the  MANPRINT  .  .  .  characteristics  of  a  system  .  .  .  unless  previous  test  and 
performance  data  or  market  analysis  (information)  is  adequate  for  verifying  operational 
effectiveness  and  suitability  of  the  system"  (p.  27).  Sections  3-8  and  3-9  of  AR  602-2  also 
define  MANPRINT  requirements  for  NDI  and  product  improvements.  In  fact, 
MANPRINT  concerns  alone  can  provide  the  justification  for  a  product  improvement. 
MANPRINT  considerations  are  clearly  related  to  OWL. 

While  not  specific  to  OWL,  higher  level  documentation  calls  out  HFE  requirements 
in  all  phases  of  the  acquisition  process.  AR  602-1  specifies  HFE  requirements  throughout 
the  materiel  life  cycle  and  stipulates  that  the  HFE  program  shall  be  performed  in  accordance 
with  MIL-H-46855B,  thereby  indirectly  establishing  the  requirement  that  OWL  issues 
need  be  addressed.  Also,  program  objectives  such  as  to  ".  . .  Insure,  through  basic  and 
applied  studies  and  research  in  HFE  . . .  that  equipment  designs  and  operational  concepts 
are  compatible  with  the  capabilities  and  limitations  of  operators  and  maintenance  personnel" 
(p.  1-4)  additionally  point  toward  addressing  workload  issues. 

In  concert  with  AR  602-1  is  the  Army's  new  regulation  for  the  implementation  of 
its  MANPRINT  concept,  AR  602-2.  As  it  may  be  recalled,  MANPRINT  is  an  umbrella 
concept  that  encompasses  HFE,  manpower,  personnel,  training,  health  hazard  assessment. 
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and  system  safety.  As  such,  it  (AR  602-2)  assumes  the  responsibility  for  coordinating  the 
requirements  of  its  constituent  domains.  Thus,  MANPRINT  policy  provides  that  HFE 
Analysis  will  be  prepared  in  accordance  with  AR  602-1  on  all  Army  major,  designated 
acquisition,  and  in-process  review  (IPR)  programs.  Also,  like  AR  602-1,  AR  602-2 
addresses  the  concept  of  workload  without  specifically  using  the  term  [e.g.,  "... 
Analyses  of  the  work  environment  also  includes  consideration  of  the  physical  and  cognitive 
demands  on  personnel ..."  (p.  3),  and "...  Ensure  through  studies  and  analyses  and 

basic  and  applied  research  (human  factors  engineering, . . . )  that  equipment  designs  and 
operational  concepts  are  compatible  with  the  limits  of  operators  and  maintainers  defined  in 
the  target  audience  descriptions  ..."  (p.  3-4)].  Thus,  specification  of  MANPRINT 
analysis  requirements  is  equivalent  to  specifying  HFE  and  OWL  analysis  requirements 
within  the  appropriate  problem  domain.  This  is  important  because  higher  level 
documentation,  such  as  AR  70-1  tends  to  address  issues  and  requirements  more  generically 
as  MANPRINT  issues  and  requirements.  This  leaves  the  relevant  lower  level  documents  to 
spell  these  out  the  details  for  the  six  application  areas  of  MANPRINT. 

This  review  altogether  lead  to  the  conclusion  that  attention  to  OWL  concerns  is 
currently  required  for  all  Army  materiel  acquisition  programs.  In  part,  this  conclusion  is 
not  immediately  obvious  because  at  upper  levels  of  acquisition  process  requirements,  the 
OWL  issues  are  subsumed  under  the  more  general  requirements  for  MANPRINT/HFE. 
Also  coupled  with  this  is  the  fact  that  until  recently,  with  the  advent  of  programs  like 
MANPRINT,  HFE  issues  have  not  always  received  their  proper  attention.  This  may  be 
especially  true  for  operator  workload  analysis  which  have  not  been  as  well  developed  as 
other  more  traditional  analyses  of  HFE. 

2.4. 1.2  Military  Specification  MIL-H-46855B  and  Associated  Data  Item 
Descriptions 

The  purpose  of  Military  Specification  MIL-H-46855B  is  to  establish  and  define 
requirements  for  applying  human  engineering  to  the  development  and  acquisition  of 
military  systems,  equipment,  and  facilities.  The  human  engineering  effort  consists  of 
analysis,  design  and  development,  and  test  and  evaluation.  An  outline  of  the  detailed 
requirements  are  given  in  Table  2.4. 1-2. 
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Table  2.4. 1-2  Outline  of  MIL-H-46855B  Requirements 


3.  REQUIREMENTS 

3.1  General  Requirements 

3.1.1a  Analysis 

3.1.1b  Design  and  Development 
3.1.1c  Test  and  Evaluation 

3.1.2  Human  Engineering  Program  Planning 

3.1.3  Nonduplication 

3.2  Detail  Requirements 

3.2.1  Analysis 

3.2. 1.1  Defining  and  Allocating  System  Functions 

3.2.1. 1.1  Information  Flow  and  Processing  Analysis 

3.2.1. 1.2  Estimates  of  Potential  Operator/Maintainer 

Processing  Capabilities 

3.2.1. 1.3  Allocation  of  Functions 

3.2. 1.2  Equipment  Selection 

3.2.1.3  Analysis  of  Tasks 

3.2.1.3.1  Gross  Analysis  of  Tasks 

3.2. 1.3.2  Analysis  of  Critical  Tasks 

3.2. 1.3.3  Workload  Analysis 

3.2.1.3.4  Concurrence  and  Availability 

3.2. 1.4  Preliminary  System  and  Subsystem  Design 

3.2.2  Human  Engineering  in  Equipment  Detail  Design 

3.2.2. 1  Studies,  Experiments  and  Laboratory  Tests 

3.2.2. 1.1  Mockups  and  Models 

3.2.2. 1.2  Dynamic  Simulation 

3.2.2.2  Equipment  Detail  Design  Drawings 

3.2.2.3  Work  Environment,  Crew  Stations  and  Facilities  Design 

3.2.2.4  Human  Engineering  in  Performance  and  Design 

Specifications 

3.2.2.5  Equipment  Procedure  Development 

3.2.3  Human  Engineering  in  Test  and  Evaluation 

3.2.3. 1  Planning 

3.2.3.2  Implementation 
3.2.33  Failure  Analysis 

3.2.4  Cognizance  and  Coordination 


The  analysis  section  deals  primarily  with  task  analysis,  function  allocation,  and  estimates  of 
potential  operator/maintainer  processing  capabilities.  Specifically,  it  provides: 

"3.2. 1 .3.3  Workload  Analysis  -  Individual  and  crew  workload  analysis 
shall  be  performed  and  compared  with  performance  criteria." 
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However,  no  further  information  is  given  as  to  how  to  perform  such  an  analysis 
nor  what  performance  criteria  should  be  used.  This  specific  reference  for  OWL  analysis 
consequently  comes  under  the  domain  of  Human  Factors  Engineering  (HFE).  This 
military  specification  contains  in  its  appendix  an  application  matrix  that  gives  guidelines  as 
to  what  sections  of  the  specification  should  be  applied  during  what  phases  of  the  life  cycle 
as  well  as  what  modifications  should  be  made  depending  on  the  life  cycle  phase.  The  MIL- 
H-46855B  appendix  shows  that  specific  workload  analysis  provision  is  in  effect  in  all 
phases  of  the  life  cycle. 

Data  item  description  (DID)  describes  data  and  prescribes  preparation  instructions 
for  the  data  for  the  analyses  called  out  by  MIL-H-46855.  The  series  of  DIDs  on  human 
factors  engineering  call  for  a  wide  range  of  information  —  the  DIDs  are  listed  in  -Table 
2.4. 1-3.  The  specific  DIDs  that  contain  the  requirements  for  implementation  of  this  section 
are  DI-H-7056,  Human  Engineering  Design  Approach  Document  -  Operator  (HEDAD-0), 
and  DI-H-7057,  Human  Engineering  Design  Approach  Document  -  Maintainer.  DI-H- 
7052,  Human  Engineering  Dynamic  Simulation  Plan,  while  not  referencing  Section 
3.2. 1.3. 3,  does  contain  the  specific  provision  for  the  use  of  dynamic  simulation  for 
workload  analysis.  Of  particular  interest  is  DI-H-7056  because  of  its  specific  application  to 
operators.  Basically,  the  operator-equipment  interfaces  and  the  task  analyses  results  are  to 
be  presented  in  the  HEDAD-O. 

2.4. 1 .3  Aeronautical  Design  Standard- 30 

The  Aeronautical  Design  Standard  for  Human  Engineering  Requirements  for 
Measurement  of  Operator  Workload  (ADS-30)  was  the  only  Army  document  found  that 
dealt  specifically  with  OWL.  The  proponent  is  the  U.S.  Army  Aviation  Systems 
Command,  St.  Louis,  MO.  ADS-30  "establishes  the  requirement  for  a  comprehensive, 
broadly-based  workload  assessment"  to  identify  workload  "chokepoints"  in  materiel 
systems  (i.e..  Army  aviation  systems).  This  standard  provides  for  the  workload 
assessment  to  be  carried  out  throughout  the  design  and  development  portions  of  the 
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Table  2.4. 1-3  Human  Factors  Engineering  Data  Item  Descriptions 


Number 

Title 

Dr-H-7051 

Human  Engineering  Program  Plan 

DI-H-7052 

Human  Engineering  Dynamic  Simulation  Plan 

DI-H-7053 

Human  Engineering  Test  Plan 

DI-H-7054 

Human  Engineering  System  Analysis  Report 

DI-H-7055 

Critical  Task  Analysis  Report 

DI-H-7056 

Human  Engineering  Design  Approach  Document-Operator 

DI-H-7057 

Human  Engineering  Design  Approach  Document- 
Maintainer 

DI-H-7658 

Human  Engineering  Test  Report 

DI-H-7059 

Human  Engineering  Progress  Report 

acquisition  process.  Types  of  OWL  techniques  are  discussed  and  management  methods  for 
assessment  by  contractors  are  discussed. 

2.4. 1 .4  Department  of  Defense  Dociimp.nf^ 

Three  Department  of  Defense  (DoD)  level  documents  pertaining  to  system 
acquisition  were  reviewed  for  guidance  regarding  OWL.  (These  are  listed  in  Table  2.4.1- 
4.)  DoD  Directive  (DoDD)  5000.1,  the  first  of  these,  presents  DoD  acquisition  policy  for 
major  systems  or  major  modifications  to  existing  systems.  Broad  guidance  for  technical 
issues  is  mcluded  in  the  list  of  acquisition  management  principles  and  objectives.  The  most 
applicable  prmciple  to  OWL  issues  is  that  improved  readiness  and  sustainability  are  primary 
objectives,  with  operational  suitability  of  equal  importance  to  operational  effectiveness. 
Operational  effectiveness  is  the  overall  degree  of  mission  accomplishment  of  the  system. 
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Operational  suitability  is  the  degree  to  which  the  system  can  be  placed  satisfactorily  in  field 
use,  with  consideration  given  to  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors, 
manpower  supportability,  logistic  supportability  and  training  requirements.  Operational 
suitability  includes  the  ability  of  the  soldier  to  operate,  maintain  and  support  the  system, 
which  would  include  OWL. 


Table  2.4. 1  -4  Department  of  Defense  Documents 


Document 

Subject 

Date 

DoD  Duective 

5000.1 

Major  Systems  Acquisitions. 

% 

12  Mar  86. 

DoD  Instruction 

5000.2 

Major  System  Acquisition  Procedures 

12  Mar  86. 

DoD  Directive 

5000.3 

Test  and  Evaluation 

12  Mar  86. 

DoD  Instruction  (DoDI)  5000.2  describes  the  procedures  to  implement  DoDD 
5000.1.  Workload  issues  are  never  specifically  addressed,  but  might  become  topic  areas 
included  in  a  program  review  or  in  the  program  documents  used  in  support  of  DoD  level 
decisions.  DoDD  5000.3  provides  policy  and  guidance  for  test  and  evaluation  in  DoD  and 
provides  guidance  for  the  TEMP.  Again,  specific  issues  are  not  addressed  in  this  directive, 
although  "use  of  properly  validated  analysis,  modeling  and  simulation  is  strongly 
encouraged,  especially  during  early  development  phases..."  (p.  3).  An  important  aspect  of 
testing  and  evaluation  is  addressing  critical  issues  that  have  been  identified  or  may  arise 
throughout  the  MAP.  The  DoD  Directives  (5000.1-5000.3)  do  not  address  specific  OWL 
but  provide  high  level  policy  that  operational  suitability  is  important  and  test  and  evaluation 
should  address  important  issues. 
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2.4. 1 .5  Other  Sources 


Table  2.4. 1-5  lists  several  further  Army  sources  which  that  address  OWL.  The 
MBL-STD  T472C,  the  first  of  these,  is  the  basic  military  standard  for  human  engineering 
design  criteria.  The  general  requirements  for  equipment  design  include  the  principle 
"Design  shall  be  such  that  operator  workload,  accuracy,  time  constraint,  mental  processing 
and  communication  requirements  do  not  exceed  operator  capabilities"  (p.  13).  Similarly, 
software  is  to  be  designed  to  minimize  task  complexity  and  memorization.  Operator 
response  times  will  be  with  operational  task  limits  (p.  242).  The  term  "workload"  appears 
in  one  other  place;  on  p.  63,  it  is  stated  that  the  "distribution  of  work  load"  should  be  such 
that  none  of  the  operator  limbs  are  overburdened.  This  is  workload  in  the  functional 
allocation  sense.  Interestingly,  "workload"  does  not  appear  in  the  index.  MIL-STD- 
1472C  is  a  basic  document  where  designers  might  look  for  information,  but  does  not 
provide  any  information  or  suggestions  specifically  addressing  OWL  (with  the  few 
exceptions  noted  above). 

Two  sources  were  provided  to  us  by  individuals  at  TECOM.  The  TECOM 
Pamphlet  602-1  (Vol.  1),  in  particular,  describes  how  to  design  subjective  opinion  tests 
and  was  identified  by  those  individuals  as  "what  was  used  to  assess  workload  in  technical 
tests."  The  second  source  was  the  Test  Operating  Procedure  (TOP)  1-2-610  which 
provides  detailed  design  criteria  against  which  to  test  equipment.  One  of  the  procedures 
included  is  a  Workload  Assessment  (p.  131)  which  suggests  a  time-line  analysis  and 
supplementing  these  observations  with  subjective  questions.  A  Workload  Assessment  form 
is  included  with  headings  such  as  critical  task,  time  required,  additional  tasks  conducted 
simultaneously,  effects  of  time  delays  in  task  completion,  overload  problems  and  underload 
problems.  This  is  addressing  "workload"  in  the  context  of  sharing  or  consolidating  the 
tasks  to  be  accomplished. 
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Table  2.4. 1-5  Other  Army  OWL  Sources 


Document  Title _ Date _ 

MIL-STD-1472C 

Human  Engineering  Design  Criteria  2  May  8 1 

for  Military  Systems,  Equipment  and 

Facilities 

ADS-30  Human  Engineering  Requirements  17  Nov  86 

for  Measurements  of  Operator 
Workload 


TECOM  Pam  602-1 

Questionnaire  and  Interview  Design  25  Jul  75 
(Voll)  (Subjective  Testing  Techniques) 

TOP  1-2-610  Hiunan  Factors  Engineering  Data  20  Nov  83 

Guide  for  Evaluation  (HEDGE) 


Several  documents  originally  identified  as  relevant  were  unavailable  for  the  present 
review  because  they  are  under  revision  or  out  of  stock  at  the  document  distribution  center. 
The  documents  include: 

•  MIL-HDBK-759  "Human  Factors  Engineering  Design  for  Army 
Materiel" 

♦  AR  10-41  "Organization  and  Functions,  U.S.  Army  Training  and 
Doctrine  Command" 

♦  AR  15-14  "Systems  Acquisition  Review  Council  Procedures 

•  MIL-STD-1388-1/2  "Logistic  Support  Analysis  "/"Logistic  Support 
Analysis  Record" 

Efforts  to  obtain  and  review  these  particular  documents  as  well  as  identify,  obtain 
and  review  other  relevant  documents  to  enhance  our  understanding  of  OWL  requirements 
will  continue.  It  is ,  however,  believed  that  the  present  review  has  essentially  revealed  the 
status  of  OWL  in  the  Army  MAP. 
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2.4.2  Terminology 


The  term  "workload"  carries  a  multitude  of  meanings  within  the  military 
community.  For  example,  HARDMAN-type  manpower,  personnel  and  training  (MPT) 
analyses  specify  workload  and  workload  analysis  for  system  operators  and  maintainers. 
Within  this  context,  however,  workload  is  defined  as  the  number,  frequency  and  durations 
of  activity-based  tasks,  performed  by  a  specific  number  of  personnel  of  particular  MOS's, 
skill  levels  and  paygrades.  The  output  of  workload  analyses  are  numbers  and  grade 
qualifications  of  personnel  necessary  to  operate  and/or  maintain  a  system.  This 
performance/manhour  based  interpretation  is  much  more  restricted  than  that  which  is  taken 
here  or  that  is  necessary  to  fully  address  OWL  issues.  Within  other  contexts,  it  is  clear  that 
workload  does  not  refer  to  cognitive/physical  underload  or  overload,  but  rather  to  -task- 
based  manning  considerations.  It  seems,  then,  that  care  must  be  taken  to  clearly  discuss 
exactly  what  is  being  discussed  when  using  terms  like  "workload"  and  "workload 
analysis".  Certainly,  manpower  considerations  are  closely  tied  to  potential  "cognitive 
overload"  (see  section  2.4.3),  but  they  are  different  and  should  be  clearly  differentiated. 
Care  must  also  be  taken,  as  intended  here,  when  addressing  a  military  audience  to  insure 
that  the  proper  framework  for  discussing  perceptual,  cognitive,  psychomotor  workload,  is 
established  up  front. 

2.4.3  MANPRINT 


The  Army  Manpower  and  Personnel  Integration  (MANPRINT)  initiative  focuses 
on  the  soldier-in-the-loop  and  front-end  analysis  in  the  acquisition  process.  As  noted 
earlier,  MANPRINT  seeks  to  integrate  six  areas  that  are  concerned  with  the  soldier  into 
the  materiel  acquisition  process  so  that  soldier  needs  and  abilities  are  considered  early  in  the 
process.  The  six  areas,  called  domains,  are  Manpower,  Personnel,  Training,  Human 
Engineering,  Health  Hazards  Assessment,  and  System  Safety. 

2.4.3. 1  Interrelationships  between  OWL  and  the  Domains 

Manpower,  personnel  and  training  (MPT)  are  critical  areas  in  the  Army  MAP. 
Manpower  is  concerned  with  force  structure  and  deals  with  how  many  people  and  of  what 
Military  Occupational  Speciality  (MOS)  are  needed  to  operate,  maintain,  and  support 
materiel.  These  are  sometimes  referred  to  as  the  "spaces."  Personnel  issues  deal  with  the 
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kind  of  people  needed  to  operate,  maintain,  and  support  materiel.  People  in  this  context 
are  recognized  as  possessing  different  levels  of  intelligence  and  skill,  as  well  as  different 
personality  attributes.  Personnel  issues  are  sometimes  referred  to  as  the  "faces",  implying 
the  individual  characteristics  of  the  soldier.  Training,  of  course,  is  the  instruction  of  the 
soldier  in  specific  skills  and  procedures  needed  to  perform  necessary  tasks.  Training  is 
done  in  schools  and  in  units  and  many  methodologies  are  used.  Each  of  these  areas  must  be 
addressed  in  the  development  of  Army  requirements  and  are  not  synonymous  with  OWL. 
However,  there  are  interrelationships  between  OWL  and  MPT  that  should  be  kept  in  mind 
as  this  effort  proceeds. 

One  relationship  between  manpower  and  OWL  is  the  term  "workload"  (as 
discussed  in  Section  2.4.2).  For  many  tasks,  it  is  not  inappropriate  to  conclude  that  if 
there  is  too  much  for  one  person  to  do  in  a  certain  amount  of  time  to  specific  criteria  levels, 
then  having  two  people  do  the  job  will  take  care  of  the  problem.  However,  this  relatively 
simple  addition  of  more  people  (assuming  that  the  additional  people  with  the  needed 
abilities  are  available)  will  not  solve  every  problem.  The  process  of  perceiving  and 
processing  must  ultimately  rest  on  single  operators  in  many  circumstances.  Adding  another 
person  in  this  case  would  not  help. 

The  distinction  between  between  manpower  and  OWL  concerns  may  be  made  by 
questioning  (1)  whether  the  task(s)  that  are  creating  the  "workload"  are  of  the  kind  that  can 
solved  by  just  adding  another  person  or  (2)  is  the  nature  of  the  task  such  that  it  must  be 
done  by  an  individual  and  it  requires  too  much  in  too  short  a  time  period. 

Personnel  issues  are  concerned  with  the  individual  characteristics  of  the  soldiers. 
The  interrelationship  between  OWL  and  personnel  issues  involve  such  areas  as  the  trade 
offs  between  intelligence  and  "quality"  as  identified  by  the  AS  VAB  (i.e.,  mental  categories 
I-IV)  and  the  degree  of  soldier  perceptual,  cognitive,  or  psychomotor  loading.  Trade  offs 
may  need  to  be  identified  depending  on  the  types  of  soldiers  available  because  of  this 
interaction  of  personnel  characteristics  and  system  design. 

Training  is  another  area  with  which  OWL  is  interrelated.  Increased  training  gives 
the  soldier  knowledge,  skills,  and  practice  in  the  required  tasks.  Additional  training  may  be 
and  is  frequently  treated  as  the  solution  to  overcome  inadequate  performance.  However, 
training  often  may  not  be  effective  in  reducing  workload,  or  a  cost  effective  way  of  and 
enhancing  performance  (Hart,  1986).  In  order  to  adequately  control  the  more  cognitive 
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workload,  there  may  need  to  be  trade  offs  made  between  training  and  the  quality  of  soldier 
chosen  as  the  operator.  Another  alternative  is  to  affect  the  hardware  design  as  part  of  HFE. 

Human  Factors  Engineering  (HFE)  is  concerned  with  the  engineering  design  of 
equipment  and  the  soldier-machine  interface  so  that  system  performance  (including  the 
human  element)  is  maximized.  OWL  issues  are  interrelated  with  HFE  in  the  design  of  the 
equipment,  specifically  the  soldier-machine  interface  design.  If  an  interface  has  been 
designed  well,  the  ease  with  which  the  operator  can  perceive  information  or  perform 
motor  tasks  may  be  optimal,  thereby  reducing  workload.  An  unthoughtful  and  poorly 
designed  interface  may  be  the  major  factor  in  creating  a  workload  intensive  task.  Human 
factors  engineering  solutions  should  certainly  be  among  the  first  pursued  when  overload 
problems  are  identified. 

Health  hazard  assessment  (HHA)  is  concerned  with  any  condition  inherent  to  the 
use  of  equipment  that  may  cause  degradation  of  job  performance,  chronic  disability  or 
death.  Health  hazards  include  toxic  substances,  vibration,  noise,  temperature  extremes  and 
psychological  stressors.  Although  this  last  area  is  not  universally  considered  a  health 
hazard,  psychological  stressors,  such  as  confined  spaces,  isolation,  sleep  deprivation,  and 
sensory/cognitive  overload,  may  cause  serious  degradation  or  chronic  disability  in  job 
performance.  There  is  currently  no  overall  Army  program  addressing  these  psychological 
stressors  from  a  HHA  perspective.  However,  health  hazard  assessors  try  to  identify 
potential  problems  early  in  the  acquisition  process  and  raise  a  flag  that  this  issue  must  be 
considered  as  the  MAP  continues  (LTC  B.  Leibrecht,  USAARL,  personal  communication, 
30  April  1987).  OWL  should  be  an  issue  from  the  HHA  perspective  in  the  acquisition 
process. 

System  safety  (SS)  is  concerned  with  identifying  and  eliminating  or  reducing  the 
risks  associated  with  system  (particularly  hardware)  characteristics  that  may  cause  injury  or 
death.  The  results  of  hardware  failure  (e.g.,  electrical  shorts  or  restraint  harnesses 
breaking)  are  of  particular  concern.  OWL  issues  are  related  to  the  safety  of  the  system  only 
and  to  the  extent  these  risks  intrude  and  occupy  the  operator. 

It  can  be  concluded  from  the  above  discussions  that  operator  workload  is  related  to 
all  six  areas  of  MANPRINT.  It  is  not  clearly  synonymous  with,  nor  falls  under  the 
specific  purview  of,  any  particular  domain.  However,  the  interrelationships  between 
OWL  and  the  MANPRINT  domains  are  important  considerations  in  developing  system 
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requirements  and  design.  Considerations  of  these  interrelationships  will  identify  MPT, 
HFE,  HHA  or  SS  trade-offs  that  may  be  made  in  an  effort  to  control  OWL  as  system 
requirements  and  design  are  defined. 

As  specified  by  the  Army  in  its  training  courses  on  MANPRINT,  there  are  several 
tools  to  be  employed  by  the  six  MANPRINT  domains.  Among  these  is  workload  analysis, 
which  the  Army  indicates  is  to  be  used  during  all  phases  of  the  acquisition  process  to 
answer  such  questions  as: 

•  Which  design  alternative  is  the  best? 

•  What  training  will  be  required? 

•  Can  operators  perform  all  functions  effectively? 

•  What  design  inadequacies  exist  that  must  be  rectified? 

These  questions  as  well  as  others  are  raised  during  the  various  phases  of  the 
acquisition  process  as  appropriate,  and  workload  estimation  and  measurement  techniques 
must  be  developed  that  can  provide  timely  answers. 

2.4.3.2  System  MANPRINT  Management  Plan 

The  System  MANPRINT  Management  Plan  (SMMP)  is  the  "planning  and 
management  guide  and  an  audit  trail  to  identify  tasks,  analyses,  trade-offs,  and  decisions 
that  must  be  made  to  address  MANPRINT  issues  during  the  materiel  development  and 
acquisition  process"  (AR  602-2,  p.  12). 

The  SMMP  is  initiated  very  early  in  the  acquisition  process  by  the  combat  or 
training  developer  and  requires  consideration  of  concerns  and  questions  that  may  affect 
soldier  performance  in  Army  equipment  This  is  an  appropriate  and  logical  place  to  include 
OWL  concerns  and  has  the  important  attribute  of  being  initiated  at  the  very  outset  of 
materiel  requirements  development.  The  SMMP  is  to  be  started  prior  to  the  program 
initiation,  and  most  likely  will  be  developed  concurrently  with  the  O  &  O  Plan.  Even  at  this 
point,  OWL  concerns  can  be  raised  and  methods  to  answer  questions  and  address 
concerns  can  be  suggested. 

The  format  for  the  SMMP  is  given  in  Appendix  B  of  AR  602-2.  The  five  major 
sections  include: 

•  a  summary  of  MANPRINT  strategy 
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•  a  description  of  the  proposed  system,  with  the  acquisition  strategy, 
agencies  involved  and  any  existing  guidance  also  described 

•  a  description  of  the  MANPRINT  strategy  including  the  objectives,  data 
source  availability,  and  planned  MANPRINT  analyses 

•  any  issues  or  areas  of  concern  which  have  been  identified 

•  the  tabs,  which  include  list  of  data  sources,  MANPRINT  milestone 
schedule,  task  description,  questions  to  be  resolved,  and  list  of  all 
organizations  with  which  the  SMMP  must  be  coordinated. 

Although  the  regulation  requiring  the  SMMP  is  new  (17  April  1987),  some 
guidance  is  available  through  the  SMMP  Procedural  Guide  (1986).  It  is  expected  that  as 
more  experience  is  gained  by  those  who  prepare  the  SMMP,  the  SMMPs  will 
increasingly  address  issues  in  the  MANPRINT  areas  and  provide  a  useful  management 
plan  to  control  factors  such  as  OWL. 

The  SMMP  has  several  sections  which  provide  opportunities  to  address  OWL 
concerns  early  and  throughout  the  MAP.  In  particular,  the  Concerns  section  (paragraph  4) 
is  the  place  to  discuss  any  identified  issues  in  the  system  development.  These  concerns  are 
those  that  should  be  monitored  throughout  the  MAP.  Further,  the  Questions  to  be 
Resolved  (Tab  D)  are  the  detailed  questions  that  need  to  be  answered  to  address  the 
concerns  identified  in  Paragraph  4.  These  questions  should  be  detailed  and  specific.  In 
some  ways,  development  of  the  Tab  D  questions  will  lead  to  the  analyses  that  need  to  be 
done  in  order  to  obtain  sufficient  information  to  answer  the  questions  (these  analyses  are  to 
be  presented  as  part  of  Paragraph  3b  as  well  as  the  identification  of  predecessor  or 
reference  systems  and  what  kind  of  data  is  expected  to  be  available  for  use).  Tab  A  is  also 
identified  as  the  place  to  list  all  potential  data  sources  in  all  the  MANPRINT  domains  and 
should  also  include  those  relevant  to  OWL.  A  description  of  the  tasks  to  be  done  in  support 
of  MANPRINT  efforts  are  to  be  presented  in  Tab  C.  These  descriptions  include  the 
rationale,  resources  needed,  time  to  complete,  and  responsible  agencies. 

Clearly,  if  an  awareness  of  and  sensitivity  to  OWL  issues  can  be  developed  by 
those  preparing  the  SMMPs,  then  their  format  should  provide  the  means  to  surface  broad 
concerns  about  workload  issues  and  well  as  the  specific  questions  that  need  to  be 
investigated  in  order  to  adequately  address  the  stated  concerns.  The  identification  of 
predecessor  system  and  data  list  will  directly  affect  the  types  of  OWL  predictive  and/or 
evaluative  assessments  that  can  be  conducted.  Similarly,  it  will  affect  the  timeliness  with 
which  such  assessment  can  be  performed  in  the  sense  that  well-documented  data  sources 
and  known  availability  will  facilitate  its  gathering  and  application. 
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Any  key  OWL  issues  or  concerns  should  also  be  included  in  the  summary 
(Paragraph  1)  which  presents  an  overview  of  MANPRINT.  The  summary  will  be  the 
portion  most  often  read  by  decision  makers  and  will  give  visibility  to  the  key  issues.  The 
status  of  key  issues  can  be  monitored  and  managed  as  the  SMMP  is  continually  updated 
with  current  information. 

Another  aspect  of  the  SMMP  that  is  of  interest  in  the  control  of  OWL  is  the  role  of 
the  MANPRINT  Joint  Working  Group  (MJWG).  Although  the  SMMP  is  the 
responsibility  of  the  Combat  Developer  (i.e.,  TRADOC),  it  is  to  be  developed  in 
conjunction  with  the  Materiel  Developer  (i.e.,  AMC).  The  MANPRINT  Joint  Working 
Group  (MJWG)  is  the  group  of  people  that  work  together  to  create  the  SMMP  and  includes 
representatives  from  the  Combat  Developer,  the  Materiel  Developer,  and  'other 
organizations  that  are  involved.  The  SMMP  should  consequently  have  inputs  from  all 
interested  organizations  and  they  can  play  a  part  in  the  lists  of  concerns,  questions,  and 
tasks  to  be  accomplished. 

2.4.4  Identified  Army  Protects 

During  the  document  review,  other  procedures  and  analyses  that  have  been 
developed  for  the  Army  and  that  may  be  useful  in  this  effort  were  identified.  As  we 
continue  our  investigations,  existing  procedures  or  information  that  can  be  used  in 
workload  analyses  will  be  utilized  to  the  fullest  extent  possible.  Some  of  the  identified 
sources  and  the  potential  use  in  OWL  assessment  are  delineated  in  the  following. 

2.4.4. 1  Early  Comparability  Analysis  (ECA) 

The  Early  Comparability  Analysis  (ECA)  Procedural  Guide  (1986)  summarized 
that  "the  ECA  methodology  is  based  on  a  'lessons  learned'  approach  to  the  design  of  a 
conceptual  system"  (p.  1).  For  ECA,  a  predecessor  (or  reference)  system  is  identified 
with  which  to  compare  the  conceptual  system.  Relevant  MOSs  are  identified,  task  lists  for 
those  MOSs  are  obtained  (or  derived  if  not  available).  Subject  Matter  Experts  (SMEs)  are 
also  consulted  to  assign  "difficulty"  ratings  to  tasks  using  six  task  criteria;  percent 
performing,  task  learning  difficulty,  task  performance  difficulty,  frequency  rate,  decay  rate, 
and  time  to  train.  Those  tasks  costly  in  MPT  resources  are  identified  and  solutions  are 
proposed  to  eliminate  or  reduce  the  cost  of  these  "high  drivers." 
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EGA  uses  a  combination  of  several  analytic  techniques  to  identify  MPT  resource¬ 
intensive  tasks  (comparison,  SMEs,  task  analysis) .  As  discussed  previously,  MPT  issues 
are  interrelated  with  OWL  issues  and  their  treatment  in  the  context  of  an  analysis  already  in 
place  can  yield  OWL  information  very  early  in  the  MAP.  This  could  enhance  the 
development  of  OWL  analyses  as  EGA  is  designed  for  use  by  combat  developers  as  an  in- 
house  tool,  therefore  it  is  not  intended  to  be  a  highly  sophisticated  tool  for  use  only  by 
technical  experts.  OWL  estimations  have  been  previously  made  based  upon  comparability 
(e.g.,  Shaffer  et  al..  1986),  SME  (Zachary,  1980),  and  task  analysis  (e.g.,  Stone  et  al.. 
1985). 


2.4.4.2  Hardware  vs.  Manpower  (HARDMAN) 

The  HARDMAN  Gomparability  Analysis  is  a  "structured  approach  to  the 
determination  of  the  Manpower,  Personnel  and  Training  (MPT)  requirements  of  a  weapon 
system  in  the  earliest  phases  of  its  development"  (Mannle,  Guptill,  and  Risser,  1985). 
HARDMAN  is  primarily  an  MPT  analysis  and  sophisticated  comparison  methodology  is 
used  to  derive  estimates  for  MPT.  It  does  produce  task  analyses  with  time-lines  that  could 
be  used  for  certain  OWL  estimation  methodologies  (e.g..  Stone  et  al..  1985).  However, 
HARDMAN  is  a  sophisticated  tool  -  currently  only  one  company  performs  the  analyses 
for  the  Army  —  and  it  is  expensive  to  do.  Therefore,  HARDMAN  wUl,  most  likely,  not  be 
available  for  use  on  all  systems  and  may  not  be  expected  to  provide  a  broad  basis  for 
predictive  analyses  of  OWL. 

2.4.4.3  Logistic  Support  Analysis  (LSA) 

The  Logistic  Support  Analysis  (LSA),  as  described  by  AR  700-127,  is  performed 
to  identify  existing  or  proposed  support  structure  and  requirements,  as  well  as  apply 
Integrated  Logistics  Support  (ILS)  and  MANPRINT  influence  in  system  design  and 
selection.  LSA  is  required  in  all  acquisition  programs.  As  part  of  the  LSA,  certain  tasks  are 
required  of  both  the  combat  and  materiel  developers.  Such  tasks  as  use  studies  (Task  201), 
comparative  analyses  (Task  203),  and  task  analyses  (Task  401)  are  required  in  accordance 
with  MIL-STD-1388.  The  LSAR  may  be  a  useful  source  of  data  for  maintainer  workload 
estimation  and  evaluation.  Further  investigation  is  needed  to  determine  what  data  would 
actually  be  available  for  use  for  assessments  of  OWL. 
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2.4.4A  Human  Resources  and  Test  Evaluation  System  (URTES) 


The  Human  Resources  and  Test  Evaluation  System  (HRTES)  (Kaplan,  Crooks, 
Sanders  and  Dechter,  1984)  is  a  set  of  procedures  designed  to  assist  a  test  planner  to 
evaluate  operator  and  maintainer  performance  in  an  operational  test  of  an  Army  system.  The 
HRTES  procedure  takes  the  test  planner  through  a  series  of  steps  from  general  system 
issues  to  the  highly  specific  human  performance  issues.  For  example,  a  set  of 
considerations  would  be  to  1)  define  mission  performance  2)  define  performance 
conditions  such  as  weather,  3)  identify  system  performance  issues  and  criteria  and  then  4) 
identify  human  tasks  and  performance  criteria  for  each  task.  Interspersed  through  this 
analysis  is  the  reminder  of  the  need  for  planning  and  for  identifying  the  techniques  for 
measuring  each  of  the  criteria.  The  HRTES  procedures  have  potential  utility  for  OWL 
assessment  with  regard  to  identifying  and  defining  system  characteristics  for  use  in  the 
selection  of  individual  techniques. 

2.4.4..5  MANPRINT  Data  Base 

A  MANPRINT  data  base  is  currently  under  development  at  the  U.S.  Army  Materiel 
Readiness  Support  Activity.  The  main  purpose  of  the  data  base  is  to  organize  MPT,  HFE, 
HHA  and  SS  data  for  use  in  comparative  analyses.  The  data  base  will  contain  historical 
data  organized  by  end  item.  The  MPT  portion  will  contain  such  items  as  RAM  data,  MOSs 
related  to  the  end  item,  manhours  and  tasks.  The  HFE,  HHA,  and  SS  area  will  be 
addressed  in  more  of  a  "lessons  learned"  approach,  with  any  problems  being  identified  and 
solutions  (if  any)  given.  The  plan  is  to  have  the  programming  of  the  data  base  on  line  by 
4th  Quarter,  FY  88.  However,  it  may  be  2  or  3  years  before  the  loading  of  the  data  is 
complete  and  the  data  base  is  accessible  to  outside  users. 

A  key  problem  in  the  data  base  development  is  the  availability  of  operator  data.  So 
far  only  the  maintainer  is  fully  documented,  primarily  because  generic  functions/subtasks 
are  more  easily  defined  for  maintainers.  It  is  much  more  difficult  to  define  generic 
functions/subtasks  for  operators  (Mr.  G.  Tarber,  USAMRSA,  personnel  communication, 
8  May  1987). 

As  the  data  base  is  further  developed  and  becomes  accessible,  it  may  provide  a 
good  source  of  data  for  workload  and  comparability  analyses.  Currently,  however, 
operators  are  not  addressed  and  there  is  not  a  firm  plan  on  how  to  do  so. 
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2.4.4.6  Summary 


These  procedures  have  been  identified  as  potential  sources  of  information  or  data 
for  use  in  OWL  assessments.  The  descriptions  of  the  procedures  and  how  they  may  be 
used  for  OWL  assessment  are  only  preliminary  judgments.  Additional  methods  and 
procedures  as  well  as  application  potentials  may  be  expected  to  evolve  because  of  the 
increasing  interest  in  programmatic  effort  (e.g.,  MANPRINT). 


2.4.5  Conclusions  and  Recommendations 

A  number  of  conclusions  can  be  drawn  from  the  document  review.  Some  of  these 
conclusions  will  have  an  impact  on  the  future  tasks  of  the  OWL  project  in  directing  how  the 
guidance  for  OWL  assessment  are  written  (i.e.,  the  handbooks).  The  major  conclusions 
and  recommendations  are: 

•  In  general,  there  is  a  void  in  Army  documents  on  the  topic  of  OWL. 

•  OWL  assessment  is  required  (via  MIL-H-46855),  but  indirectly  through 
the  areas  of  HFE  and  MANPRINT. 

•  There  do  not  seem  to  be  any  inconsistencies  regarding  OWL,  primarily 
because  there  isn't  much  available. 

•  The  intent  to  consider  the  soldier  in  materiel  acquisition  is  clear  in  high 
level  DoD  Directives. 

•  Clear  distinctions  between  types  of  workload  must  be  made. 

•  Effort  should  be  made  to  make  use  of  existing  projects  as  much  as  is 
appropriate. 

•  ASAP,  NDI  and  product  improvement  strategies  will  make  early 
attention  to  OWL  even  more  critical. 

•  MANPRINT  provides  a  framework  on  which  to  build  and  provides 
places  to  address  OWL  issues  (e.g.  the  ROC  and  the  SMMP). 

•  The  MANPRINT  portion  of  the  ROC  is  an  appropriate  place  to  address 
soldier-in-the-loop  requirements  (e.g.,  OWL)  for  the  new  equipment. 

•  The  SMMP  would  be  a  useful  vehicle  to  focus  attention  on  potential 
OWL  problems  and  devise  plans  to  address  OWL  throughout  the  MAP. 

The  review  of  the  documents  provided  a  useful  means  to  understand  the  current 
Army  requirements  and  how  issues  concerning  operator  workload  are  addressed.  The  lack 
of  specific  guidance  was  not  really  surprising  --  there  has  been  greater  awareness  in  recent 
times  because  of  the  technological  advances  being  included  in  Army  materiel.  The  identified 
lack  emphasizes  the  timeliness  and  importance  of  the  current  OWL  project  effort. 
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3.  ASSESS  USER  NEEDS 


3.1  Introduction 


In  the  effort  to  fully  understand  the  Army  MAP  and  how  OWL  issues  are 
addressed,  a  review  of  written  documents  was  conducted  as  described  in  Section  2 
subsequent  to  beginning  the  interviews.  A  better  understanding  of  the  process  and  issues 
was  obtained  by  talking  to  the  people  who  are  actually  involved  in  the  process.  The 
purposes  were  to  further  our  understanding  of  the  Army  MAP  and  learn  how  OWL  is 
currendy  considered.  These  discussions  provided  the  opportunity  to  identify  the  concerns 
of  the  Army  community  with  respect  to:  workload  (and  related  items  they  cared  to  share); 
what  guidance  or  tools  would  be  most  helpful  to  them;  and  how  things  actually  worked  as 
opposed  to  the  written  descriptions  that  had  been  reviewed. 

This  section  will  describe  our  approach  to  obtaining  information  about  Army  user 
needs  and  will  report  the  information  obtained.  With  the  exception  of  the  information 
received  via  the  questionnaires  (see  Section  3.4),  the  information  presented  has  been 
obtained  from  discussions  and  has  been  integrated  to  reflect  general  concerns  and 
suggestions,  rather  than  identifying  specific  users  and  their  comments.  This  section 
overviews:  our  approach;  user  concerns  as  expressed  in  discussions  and  questionnaires; 
user  suggestions  concerning  the  consideration  of  OWL  during  the  MAP;  as  well  as  the 
handbooks  to  be  produced  during  this  contract  effort.  Finally,  plans  for  follow-up 
assessment  of  user  needs  will  be  discussed. 

3.2  Approach 


3.2. 1  Army  Organizations  Visited 

The  Army  organizations  with  whom  we  spoke  and  the  date(s)  visited  are  presented 
in  Table  3.2. 1-1.  In  addition,  the  DoD  HFE  Test  and  Evaluation  Technical  Advisory  Group 
was  briefed  on  7-8  Apr  87  at  NASA-Langley,  VA.  Researchers  at  NASA-Langley  also 
briefed  us  regarding  their  recent  and  continuing  work  in  the  area  of  mental-state  estimation. 
Their  work  relates  directly  to  workload  assessment  and  will  be  included  in  our  Task  3 
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Table  3.2. 1- 1  Army  Organizations  Visited 


Date _ Organization 


1  Apr  87 

U.S.  Army  Materiel  Command 

Alexandria,  VA 

1  Apr  87 

U.S.  Army  Soldier  Support  Center  —  National 

Capitol  Region 

Alexandria,  VA 

2  Apr  87 

U.S.  Army  Operational  Test  and  Evaluation  Agency 

Falls  Church,  VA 

6  Apr  87 

Department  of  the  Army  Armored  Family  of  Vehicles 

Task  Force 

Ft.  Eustis,  VA 

16  Apr  87 

U.S.  Army  Test  and  Evaluation  Command 

Aberdeen  Proving  Ground,  MD 

21  Apr  87 

U.S.  Army  Aviation  Systems  Command 

St.  Louis,  MO 

23  Apr  87 

U.S.  Army  Aviation  Center 

Ft.  Rucker,  AL 

5  May  87 

U.S.  Army  Armor  School 

Ft.  Knox,  KY 

6,7  May  87 

ARE  Field  Unit  Representatives  from  Ft.  Bliss  and 

Ft.  Huachuca 

Ft.  Bliss,  TX 
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review.  Altogether,  a  wide  range  of  approximately  120  individuals  were  contacted 
including  TRADOC,  AMC,  and  other  independent  agencies. 

2-2.2 — Conduct  of  Di.scussinns 

Initial  contact  with  the  organizations  was  made  by  the  COR  and  visit  dates  and 
times  were  arranged.  Following  introductions  at  the  meeting  site,  a  25-minute  briefing 
was  given  by  a  member  of  the  Analytics  team  to  introduce  our  effort  and  explain  the 
purpose  of  the  meeting.  The  briefing  typicaUy  included: 

•  Introductory  remarks 

•  What  is  OWL? 

•  The  importance  of  OWL  in  the  Army 

•  OWL  program  objectives 

•  Our  approach:  user  inputs;  matching  model;  handbooks 

•  Candidate  System  Selection  Criteria 

Upon  completion  of  the  briefing  (or  during  as  appropriate),  the  floor  was  opened 
for  questions  and  comments.  The  discussions  focused  on  whether  OWL  is  considered  in 
their  organization;  if  so,  how  it  is  considered;  what  guidance  and  tools  are  needed;  and  any 
suggestions  for  the  products  (handbook  outlines).  The  second  focus  of  discussion  was  on 
any  emerging  systems  that  the  participants  were  aware  of  that  would  be  good  candidates  for 
validation  of  operator  workload  measures.  The  selection  of  systems  is  later  described  in 


— Questionnaire  Di.strihiitinn 

Questionnarres  and  handbook  outlines  were  distributed  at  the  conclusion  of  each 
meeting.  Participants  were  asked  to  distribute  them  to  appropriate  individuals  within  their 
organizatron  and  return  the  completed  questionnaires  to  Analytics.  A  sample  questionnaire 
IS  rncluded  rn  Appendix  A.  (The  handbook  outlines  are  discussed  in  Section  4.). 


3-3 


3.3  Army  Community  Concerns 


3  3.1  What  is  OWL? 


A  common  point  of  discussion  in  the  meetings  held  was  what  exactly  operator 
workload  implied.  There  seemed  to  be  an  understanding  of  the  problem  of  increasing 
technology  and  decreasing  personnel  resources  causing  increased  cognitive  operating 
requirements.  However,  there  does  appear  to  be  uncertainty  about  the  meaning  of  the  term 
"  workload."  The  diverse  concerns  included: 

•  Is  workload  based  on  the  number  of  tasks  to  be  performed? 

•  Does  the  workload  issue  revolve  around  the  scarcity  of  Cat  I  (i.e.,  high 
ability)  soldiers  and  the  necessity  of  using  Cat  IVs?  How  do  MOSs 
relate? 

•  Is  workload  physical ,  mental  or  both? 

•  How  is  maintainer  workload  related  to  operator  workload? 

•  How  does  workload  relate  to  MANPRINT?  Is  it  the  same  as 
MANPRINT? 

3.3.2  Organizational  Concerns 

The  Materiel  Acquisition  Process  consists  of  many  organizations  (both  government 
and  contractor)  performing  a  series  of  sequential  steps  to  achieve  the  goal  of  fielding 
effective  and  suitable  equipment  to  accomplish  a  mission.  The  sequential  nature  of  the 
process  assumes  the  previous  steps  have  been  adequately  accomplished  in  order  for  the 
next  steps  to  be  performed.  As  a  result  of  the  nature  of  the  MAP,  many  comments 
expressed  in  our  discussions  were  concerned  with  collection  and  use  of  the  system-relevant 
information  that  has  been  produced  previously  in  the  MAP.  A  second  major  area  of 
concern  was  resources.  The  resources  include  the  people  to  do  the  required  work,  the 
expertise  needed  to  do  the  work,  the  time  in  which  to  accomplish  the  work,  and  the  money 
with  which  to  pay  for  it.  Both  of  the  major  areas  of  concern  can  be  summed  up  in  the 
expression  that  was  heard  in  both  TRADOC  and  AMC  organizations  "TRADOC  doesn't  do 
its  job  up  front,  but  AMC  has  all  the  money."  This  expresses  the  frustrations  that:  (1) 
many  of  the  important  decisions  impacting  performance  and  OWL  (such  as  crew  size  and 
operational  capability)  are  determined  very  early  in  the  MAP  (as  in  the  requirements 
documents)  but  (2)  TRADOC  does  not  always  have  the  resources  or  information  to  make 
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knowledgeable  decisions  during  this  period.  These  decisions  and  requirements  are  the 
ones  that  will  subsequently  drive  the  design  and  development  of  the  system. 

There  seems  to  be  a  consensus  that  OWL  is  currently  addressed  in  a  reactive  mode. 
Only  after  it  has  been  identified  as  a  problem,  or  potential  problem,  is  any  action  taken. 
There  is  awareness  that  a  continuous  process  of  addressing  OWL  throughout  the  MAP 
would  enable  the  players  to  control  OWL  in  a  proactive  manner. 

The  people  involved  with  specific  systems  (e.g.  TRADOC  System  Managers 
(TSMs)  and  action  officers  in  Program  Manager  shops)  are  aware  of  potential  workload 
problems  but  don't  have  the  guidance,  the  expertise,  an  approach,  or  the  necessary 
financial  resources  to  adequately  address  OWL.  Increasingly,  they  are  looking  toward  the 
U.S.  Army  Research  Institute  (ARI)  and  the  U.S.  Army  Human  Engineering  Laboratory 
(HEL)  for  direction  in  areas  concerning  human  performance.  The  MANPRINT  officer  is 
also  seen  as  a  resource  for  both  human  performance  and  OWL  concerns,  however,  often 
the  MANPRINT  officer  has  not  been  on  the  job  very  long  and  does  not  have  a  great  deal  of 
experience.  The  systems  people  are  looking  for  whatever  help  they  can  get,  and  often  there 
is  only  one  ARI  or  HEL  resource  person  with  more  work  than  one  person  can  realistically 
accomplish. 

Specific  comments  were  made  concerning  the  Required  Operational  Capability 
(ROC).  A  concern,  particularly  of  testers  and  evaluators,  was  that  the  ROC  does  not 
provide  sufficient  specifics  concerning  human  performance  upon  which  to  base  test  and 
evaluation  criteria.  It  is  difficult  for  them  to  know  if  there  is  a  workload  problem,  or  a 
potential  problem,  when  there  are  inadequate  measures  of  effectiveness  (MOEs)  and  no 
specified  levels  of  performance  in  the  ROC.  Certainly  there  is  some  form  of  evaluative 
judgment  involved  in  identifying  potentially  excessive  workloads,  but  there  is  frustration  in 
the  latter  portion  of  the  MAP  in  the  test  and  evaluation  area.  Those  involved  with 
evaluation  particularly  feel  that  a  good  job  is  not  being  done  in  thinking  about  the  human 
performance  issues  in  the  front  end  of  the  MAP  in  casting  the  requirements  documents. 

A  second  comment  on  the  ROC  was  that  firm  decisions  regarding  crew  size  (i.e., 
reduced  crew  size)  are  being  included,  apparently  without  front-end  analysis  being  done 
that  would  give  information  concerning  the  reduced  crew  capability.  A  comment  made  in 
one  discussion  suggested  that  the  Army  should  design  the  capability  of  the  equipment 
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around  the  capability  of  the  crew  (with  its  size,  intellectual  and  physical  abilities 
limitations)  rather  than  fitting  the  crew  to  the  operational  wish-list  of  the  equipment. 

Another  concern  was  that  the  data  expected  to  exist  do  not  always  exist.  Because  of 
the  resource  constraints  at  the  combat  developer  schools,  such  things  as  task  lists  and 
EGAs  are  not  always  developed  in  a  timely  way  (if  ever).  Concern  was  raised  that  if  any 
additional  paperwork  or  analyses  for  OWL  are  required,  that  they  probably  wouldn't  be 
done  either.  Some  felt  that  no  matter  how  useful  and  beneficial  the  analyses  might  be,  there 
will  be  some  resistance  to  doing  them  based  on  resource  constraints  as  well  as  bureaucratic 
inertia. 

3.3.3  Major  OWL  Needs 

The  major  OWL  needs  identified  from  our  meetings  with  Army  personnel  are  the 
following: 

•  The  need  for  OWL  assessment  techniques  that  will  provide  pertinent 
OWL  information  so  that  ROCs  will  provide  a  better  framework  for  all 
subsequent  system  design  work  as  it  relates  to  human  performance 
considerations. 

•  The  need  for  OWL  assessment  techniques  that  are  not  heavily  dependent 
upon  extensive  resources  and  expertise. 

•  The  need  for  an  OWL  overview  pamphlet  that  orients  Army  personnel  to 
the  concept,  its  importance,  its  relationship  to  MANPRINT,  who  should 
be  concerned  about  it  and  when. 

3.3.4  Projected  Needs 

When  the  discussions  turned  to  anticipated  or  projected  needs  concerning 
assessment  of  operator  workload,  there  were  two  basic  categories  mentioned.  The  first  was 
the  new  emphasis  on  ASAP  and  NDI  as  the  acquisition  strategies  of  choice.  The 
streamlined  process  basically  requires  the  quality  of  development  work  to  be  done  in  a 
more  compressed  time  frame.  Therefore,  it  will  be  even  more  critical  that  the  requirements 
are  well  conceived  and  front-end  analyses  are  done  so  that  the  development  can  be 
expedited  without  running  into  design  problems.  The  NDI  strategy  presents  new  problems 
in  that  the  only  opportunity  to  ask  questions  or  obtain  data  is  in  the  market  survey  process. 
There  will  be  no  opportunity  for  testing  until  after  an  initial  purchase.  Within  the  market 
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survey,  information  can  be  requested  from  vendors,  but  there  is  no  assurance  that  the 
vendor  has  even  thought  about  the  workload  issue  or  that  any  data  pertaining  to  the  issue 
will  be  available  to  the  government.  Perhaps,  as  soldier-in-the-loop  performance  issues 
become  more  important  (e.g.,  through  emphasis  on  MANPRINT),  such  issues  and  how 
they  have  been  handled  by  the  vendors  will  be  seen  as  selling  points. 

MANPRINT  was  also  seen  as  a  driver  in  anticipated  needs  for  OWL  assessment. 
The  requirements  for  inclusion  of  MANPRINT  in  the  ROC  (AR  71-9),  the  required 
SMMP,  and  all  the  other  efforts  to  institutionalize  MANPRINT  in  the  MAP  are  seen  as 
driving  new  needs  for  data  and  analyses  concerning  soldier-in-the-loop.  The  needs  are 
particularly  seen  early  in  the  process  in  the  analytical  arena.  However,  TRADOC  plans  on 
doing  more  testing  through  its  Force  Development  Test  and  Evaluation  (FDTE)  testing 
program.  Early  testing  issues  will  become  of  more  interest  and  importance. 


3.3.5  Identified  OWL  Problems 

Our  meetings  with  Army  personnel  resulted  in  a  common  concern  being  voiced 
about  OWL.  That  is,  the  potential  for  excessive  cognitive/mental  workload  demands  being 
placed  on  operators  as  a  result  of  innovative  software  systems.  These  new  software 
systems  have  automated  many  of  the  functions  previously  done  by  operators  as  well  as 
provide  functionality  not  previously  possible  (e.g.,  new  information).  As  a  result, 
operators'  jobs  have  changed  to  being  "managers  of  information"  whereby  the 
requirements  placed  on  the  operator  are  to  manage  and  digest  the  information  provided  via 
software  systems  and  make  decisions  based  on  such  information.  This  concern  for 
excessive  cognitive/mental  OWL  was  expressed,  in  general  terms,  without  specific 
reference  to  existing  systems  per  se  but  was  foreseen  as  a  problem  with  future  developing 
systems  and  proposed  improvements  to  systems.  Section  5  in  this  report  describes  the 
prototype  systems  that  have  been  identified  as  candidates  for  the  OWL  assessment  phase 
of  this  project.  We  have  selected  prototype  systems  that  are  indicative  of  this  general 
concern  for  OWL. 
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3.3.6  Other  Workload  Initiatives 


The  issue  of  OWL  is  being  addressed  on  several  different  fronts  by  various  groups. 
These  include  Army  agencies  that  are  addressing  future  design  of  system  concepts  (Army 
HEL  Crew  Reduction  Modeling  Study)  as  well  as  theoretical  issues  concerning  OWL 
(Army  HEL  Stress  Program).  Also,  NASA  Langley  Research  Center  at  Langley,  VA  is 
exploring  OWL  in  aviation  systems  by  means  of  mental  states  estimation.  We  have  made 
contact  with  these  groups  and  have  exchanged  ideas  concerning  our  effort  as  well  as  theirs. 
Below  are  brief  descriptions  of  these  programs  objectives  as  well  as  their  stage  of 
progress. 

We  will  continue  to  identify  other  OWL  initiatives  so  to  keep  abreast  of  the,  latest 
developments  in  the  field  and  evaluate  the  applicability  of  these  initiatives  to  our  program. 

•  Army  HEL:  Crew  Reduction  Modeling  Study,  Aberdeen  Proving 
Ground,  MD.  HEL  is  the  beginning  stages  of  developing  a  computer- 
driven  mathematical  model  to  evaluate  the  feasibility,  desirability,  and 
effectiveness  of  reducing  the  size  of  combat  vehicle  crews  (e.g.,  tanks). 

They  are  in  the  process  of  selecting  an  outside  contractor  for  computer 
modeling  of  combat  crew  operation  (Sept  87). 

•  Army  HEL:  Combat  Stress  Program,  Aberdeen  Proving  Ground,  MD. 

A  multi-disciplinary  research  program  has  been  started  to  provide  basic 
human  and  weapon  systems  performance  data  under  "combat-stress" 
conditions  by  means  of  modeling  the  battlefield  conditions  that  soldiers 
are  subjected  to.  This  is  a  4-phase  program  that  is  presently  in  Phase  1. 

In  Phase  I,  efforts  are  underway  to  determine  which  "stress  indices"  are 
likely  to  reflect  the  effects  of  acute  exposure  to  stress  (i.e.,  battlefield 
concfitions)  in  test  participants  and  are  likely  to  work  best  in  simulating 
such  conditions  in  the  laboratory. 

•  NASA  Langley  Research  Center:  Mental  States  Pro^m,  Langley,  VA. 

NASA  has  just  started  a  5  year  program  whose  initial  objectives  are  to 
identify  physiological  indices  that  reflect  "mental  states"  such  as  fatigue 
and  boredom  which  have  been  shown  to  influence  operator 
performance.  Ultimately,  these  physiological  signs  will  be  monitored 
by  sophisticated  software  systems  during  aircraft  flights  in  order  to 
identify  the  points  in  time  when  software  intervention  is  needed  for 
maintaining  ffight  performance. 


3.4  Questionnaire  Results 


Nineteen  people  have  fully  responded  to  the  survey  questionnaire  to  date.  Though 
this  is  a  relatively  small  sample,  the  data  provide  a  basis  for  discussion  concerning  the 
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Army  needs  for  guidance  on  OWL.  A  copy  of  the  survey  questionnaire  can  be  found  in 
Appendix  A. 

3.4.1  Demographics 

Nine  respondents  were  civilians  who  worked  for  Army  organizations  such  as 
TRADOC,  ARI  and  HEL  while  the  remaining  ten  respondents  were  military  personnel 
who  performed  various  roles  in  the  MAP,  (e.g.,  TRADOC  assistant  system  manager). 

With  respect  to  functional  roles,  eight  respondents  indicated  they  have  MANPRINT 
responsibilities/functions  that  support  TRADOC  schools  (n=2),  TRADOC  integrating 
centers  (n=2),  and  test  and  evaluation  agencies  (n=4).  The  remaining  eleven  respondents 
were  TRADOC  project  officers  (n=2),  TRADOC  assistant  system  managers  (n=2),  and 
representatives  from  ARI  (n=3),  HEL  (n=l)  and  TRADOC  integrating  centers  (n=3). 

The  nineteen  respondents  as  a  group  represent  integral  players  in  the  MAP  who  can 
contribute  toward  addressing  OWL  during  system  development.  The  major  organization 
not  adequately  represented  in  this  sample  was  materiel  developers  (AMC).  Our  future 
plans  are  to  include  an  adequate  sample  from  AMC. 

3.4.2  Operator  Workload  (OWLl 

The  questionnaire  contained  specific  questions  on  the  importance  of  OWL  with 
respect  to  respondents’  work  (present  and  future),  the  means  they  use  now  to  address 
OWL  and  future  directions  (guidance)  needed  to  address  OWL.  The  results  were 
somewhat  suiprising. 

When  asked  how  often  the  issue  of  OWL  should  be  considered  in  their  work,  a 
majority  of  respondents  (n=13)  stated  "often"  (based  on  a  4  choice  category  scale  -  "often", 
"sometimes",  "rarely"  or  "never").  In  addition,  respondents  foresaw  a  greater  emphasis 
on  addressing  OWL  in  their  work  for  several  reasons.  Almost  all  respondents  (n=17)  saw 
changes  in  requirements  (i.e.,  MANPRINT)  as  a  force  in  directing  their  future  efforts 
toward  addressing  OWL.  Also,  a  large  majority  (n=16)  saw  "changes  in  technology" 
with  respect  to  system  innovations  as  an  important  factor  in  directing  their  work  efforts  to 
addressing  OWL.  Based  on  such  an  overwhelming  consensus  that  OWL  is  an  important 
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issue  to  be  addressed  in  their  work,  one  would  anticipate  respondents  would  employ 
similar  methodologies/tools  to  address  workload;  this  was  not  the  case.  When 
respondents  were  asked  to  state  the  specific  guidance  (documents)  they  use  to  address 
OWL,  eleven  (1 1)  people  stated  they  have  no  source  document  for  addressing  OWL  issues. 
The  remaining  eight  (8)  respondents  gave  assorted  answers  such  as  EGA,  HARDMAN, 
OWL  studies  in  journals,  and  operator  task  lists  when  available. 

When  respondents  were  asked  how  they  would  like  to  address  OWL,  answers 
varied  between  respondents.  For  example,  some  offered  no  suggestions  (n=5).  Other 
respondents  stated  that  specific  organizations  should  address  OWL  but  not  "my" 
organization  (n=3).  The  remaining  eleven  respondents  gave  individual  answers  such  as 
more  resources  devoted  to  OWL  issues,  better  defined  ROC  documents  with  respect  to 
human  performance,  videotaping  operators  performing  task  scenarios,  task  analysis, 
objective  performance  measures,  and  physiological  measures. 

When  respondents  were  asked  to  state  the  guidance  they  would  like  to  have  for 
addressing  OWL,  several  suggestions  were  offered.  These  were: 

•  Standardized  methodology/tool  for  addressing  OWL  that  requires 
minimal  resources,  it  is  non-intrusive,  in  real  time  and  characterized  as 
objective  in  nature  (n=4) 

•  Training  course  on  what  OWL  is  and  how  to  address  it  (n=2) 

•  Local  points  of  contact  (POC)  for  OWL  (n=2) 

•  OWL  Handbook  for  writing  ROC  documents  (n=l) 

•  How  to  raise  funds  to  address  OWL  (n=l). 

•  Preparation  of  a  MIL-H-46855  Workload  DID  (n=l) 

Based  on  these  findings,  it  seems  apparent  that  Army  personnel  are  concerned 
about  addressing  OWL  in  their  work,  however,  there  seems  to  be  a  lack  of  uniform 
direction  and  understanding  with  regard  to  what  OWL  is,  how  to  address  it,  and  where 
might  one  find  answers  to  these  questions. 

3.4.3  Respondents'  Work  Responsibilities  during  the  MAP 

The  survey  contained  a  series  of  questions  to  profile  the  work  done  by  Army 
personnel  as  it  relates  to  the  MAP.  Questions  pertained  to  identifying  major  work 
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responsibilities,  the  inputs  and  outputs  to  such  work  as  well  as  the  sources  of  information 
used  to  accomplish  this  work. 

Table  3.4.3- 1  lists  the  major  areas  of  responsibilities  stated  by  respondents. 
Almost  half  of  the  respondents  (n=9)  indicated  having  several  overlapping  responsibilities 
during  the  MAP,  (i.e.,  they  responded  to  more  than  one  category). 


Table  3.4.3- 1  Responsibilities/roles  during  the  MAP  that  are  held  by  respondents 

Responsibility/Role ^ Number  of  Respondents 

Define  or  review  requirements,  1 0 

standards,  criteria 

Develop  or  monitor  the  design  8 

of  emerging  system  concepts 

Design  or  monitor  the  characteristics  7 

of  early  prototype  systems 

Test  &  evaluation  of  systems  (early,  7 

mid-term,  late) 

MANPRINT  (R&D)  2 


As  seen  in  Table  3.4.3- 1,  most  respondents'  responsibilities  are  centered  in  the  early 
portions  of  the  MAP.  Their  roles  are  seen  as  critical  for  identifying  OWL  issues  early  in 
the  MAP  such  that  OWL  can  be  addressed  in  a  proactive  mode.  This  is  further  exemplified 
by  the  fact  that  the  recipients  of  their  work  are  integral  players  in  the  MAP.  Table  3.4.3-2 
shows  the  major  organizations  and  functional  roles  who  are  the  recipients  of  the  work 
accomplished  by  the  survey  respondents.  Some  respondents  indicated  having  several 
recipients  of  their  work. 
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Table  3.43-2  The  major  organizations  and  functional  roles  who  are  the  recipients  of  the 
work  produced  by  survey  respondents 

Organizations/Functional  Roles  Number  of  Respondents 

TRADOC  Schools  (e.g.,  combat  developers)  7 

Army  Materiel  Command  (e.g.,  materiel  developers)  7 
Test  &  Evaluation  Organizations  (e.g.,  OTEA)  5 

TRADOC  Headquarters  2 


In  summary,  the  respondent's  responsibilities  and  their  associated  work  are  an 
integral  part  of  the  MAP  and  serve  to  direct  all  future  work  that  occurs  during  system 
development.  But,  the  respondents  lack  standard  methodologies/tools  to  address  OWL 
issues  as  seen  by  their  answers  to  questions  about  OWL. 

3.4.4  Sources  for  Information 


With  respect  to  fulfilling  their  job  responsibilities,  respondents  listed  the  Army 
documents  as  well  as  agencies  that  they  referred  to  or  seek  guidance.  We  were  interested  in 
identifying  these  sources  in  order  to  understand  the  types  of  information  and  guidance 
sought  by  respondents.  Such  information  would  provide  insights  to  the  work  issues  that 
respondents  felt  were  important  but  lack  the  knowledge  or  experience  to  address  these 
issues  solely  by  themselves.  Table  3.4.4- 1  lists  the  major  sources  of  such  guidance.  Most 
respondents  used  multiple  sources. 
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Table  3.4.4- 1  Major  sources  for  information  and  guidance  that  are  used  by  respondents  to 
fulfill  their  job  responsibilities 


Organizations 


Number  of  Respondents 


Army  Research  Institute  (ARI) 

Army  Human  Engineering  Laboratory  (HEL) 
TRADOC  Headquarters 

TRADOC  Schools  (e.g.,  TRADOC  system  manager) 

Directorate  of  Combat  Development 

Test  &  Evaluation  Organizations  (e.g.,  OTEA) 


Documents _ 

DOD  Directive  Test  &  Evaluation 
5000.3 


AR  602-2 

MANPRINT 

11 

AR  70-10 

Test  &  Evaluation 

5 

AR71-3 

User  Testing 

4 

AR71-9 

Materiel  Objectives  &  Requirements 

4 

AR  70-8 

Personnel  Performance  &  Training 

2 

MIL  STD 

1472-C 

Human  Engin.  Design  Criteria  for 
Military  Systems,  Equipment,  & 

Facilities 

3 

HARDMAN 

(Hardware  vs.  Manpower  Compar.  Anal.)  1 

ECA 

(Early  Comparability  Analysis) 

1 

Clearly,  the  respondents'  sources  for  information  (e.g.,  ARI,  MANPRINT)  reflect 
their  concern  to  address  human-related  issues  (i.e.,  human  performance).  It  is  of  interest 
to  note  that  the  documentation  sought  for  guidance  contains  minimal  information  on  OWL. 


3.4.5  Performance  Issues  with  respect  to  the  Total  System  Development  Process 


The  survey  contained  a  series  of  questions  to  ascertain  the  different  performance 
areas  that  respondents  consider  in  order  to  do  their  job.  Table  3.4.5-1  summarizes 
participants'  responses  to  four  major  performance  areas.  Listed  are  the  number  of 
respondents  who  responded  to  a  4-choice  category  scale  ("often",  "sometimes",  "rarely", 
or  "never")  by  checking  the  "often'  category  with  respect  to  these  performance  areas. 
Table  3.4.5-2  summarizes  participants'  responses  to  major  human  performance  areas. 
Listed  are  the  number  of  respondents  who  responded  to  a  4-choice  category  scale  ("often", 
"sometimes",  "rarely",  or  "never")  by  checking  the  "often"  category  with  respect  to  these 
performance  areas. 


Table  3.4. 5-1  Number  of  respondents  who  stated  that  they  "often"  consider  these 
performance  issues  in  their  work 


Performance  Area 

Number  of  Respondents 

Total  System  Performance 

16 

Subsystem  Performance 

10 

Operator  Performance 

16 

Maintainer  Performance 

13 

Table  3.4.5-2  Number  of  respondents  who  stated  that  they  "often"  consider  these  human 
performance  areas  in  their  work 

Human  Performance  Area _ Number  of  Respondents 


Human  Factors  Engineering 

14 

Manpower 

11 

Personnel 

13 

Training:  Individual  soldiers 

16 

Training:  Unit 

8 

Safety 

11 

Health  Hazards 

10 
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It  is  quite  evident  from  these  results  (Table  3.4.5-1,  &  Table  3.4.5-2)  that 
performance  issues  (total  system  and  human  element  areas)  are  given  consideration  by 
respondents. 

3.4.6  Conclusions 

The  results  of  the  study  show  that  respondents  are  aware  and  concerned  about 
human  performance  areas.  However,  a  standardize  methodology/tool  as  well  as  a  single 
source  for  information  addressing  OWL  is  lacking.  It  also  seems  apparent  that 
respondents'  lack  of  clarity  in  their  answers  to  specific  questions  about  OWL  (e.g.,  no 
responses)  indicate  a  fuzziness  on  what  is  meant  by  OWL  and  how  to  address  it.  This 
finding  is  very  revealing  as  the  respondents  to  this  survey  have  ideal  positions  from  which 
to  address  OWL  throughout  the  MAP. 

3.5  User  S  u  ggestion  s  for  OWL  Pro  gram  in  Army 

Within  the  context  of  the  discussions  with  users,  a  number  of  suggestions  were 
given  as  to  what  the  users  thought  would  be  most  useful  to  Army  users.  Although  most  of 
the  suggestions  have  already  been  touched  upon,  a  separate  listing  was  thought  to  be 
helpful.  The  items  are  not  in  any  specific  order.  The  suggestions  are: 

•  Integrate  with  the  MANPRINT  effort  to  ensure  success. 

It  was  recognized  by  many  with  whom  we  spoke  that  to  assure 
implementation  of  any  OWL  guidance  that  was  developed,  some  type 
of  regulatory  emphasis  was  needed.  The  most  practical  suggestion  was 
that  the  MANPRINT  requirements  (e.g.,  SMMP)  be  used  as  the 
vehicles  to  address  OWL  concerns. 

•  Guidance  must  accommodate  limited  resources  and  expertise  available. 

•  Make  the  cognitive/mental  aspects  of  OWL  the  explicit  focus  of  the 
project. 

•  Capitalize  on  any  existing  information  or  data  that  relates  to  OWL. 

Information  is  generated  and  analyses  performed  to  make  decisions  in 
the  current  MAP.  Before  attempting  to  require  more  data  and 
information  generation,  be  sure  to  examine  what  is  already  available  to 
see  if  it  might  be  useful  in  answering  OWL  questions. 

•  Both  TRADOC  and  AMC  must  be  receptive  to  this  effort.  To  get  the 
most  benefit,  both  must  be  involved  in  OWL  assessment  and  control. 
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•  Create  an  associated  OWL  DID. 

There  is  an  awareness  that  contractors  will  be  in  the  correct  place  to 
gather  data  relevant  to  OWL.  In  order  to  capture  those  data,  as  well  as 
assuring  that  an  appropriate  methodology  is  being  followed,  create  a 
'  Data  Item  Description  so  that  there  is  a  means  to  obtain  data  from  the 
contractor  during  system  design  and  development. 

•  Have  awareness  that  contractors  may  be  ones  to  use  the  developed 
OWL  guidance. 

Early  analyses  that  requires  technical  expertise  or  more  man-hours  that 
internally  available  may  be  contracted  out.  Similarly,  the  realization  that 
the  RFP  is  the  most  appropriate  place  to  require  0\^  assessment. 


The  users  made  several  suggestions  regarding  the  handbooks  to  be  produced.  The 
suggestions  are: 

•  Computer-based  mode  of  presentation. 

The  users  questioned  the  use  of  written  material  (i.e.,  handbooks). 

Their  experience  has  been  that  there  is  a  tendency  to  just  put  handbooks 
on  a  shelf  and  not  use  them.  The  suggestion  was  that  guidance, 
particularly  the  predictive  and  evaluative  handbooks,  created  on  an  on¬ 
line,  interactive,  computer-based  system  might  be  better  and  easier  to 
use. 

•  Pamphlet  should  be  used  to  institutionalize  the  concept  of  OWL. 

Many  integral  players  in  the  MAP  do  not  consider  soldier-in-the-loop 
when  developing  system  concepts  and  designs.  The  pamphlet  may  be 
useful  to  draw  attention  to  the  soldiers'  performance  aspects  of  system 
performance  and  to  advise  managers  to  raise  flags  when  an  OWL 
problem  may  be  involved. 

•  Two  recommendations  should  result  from  our  guidance  for  selecting 
specific  OWL  techniques. 

Recommendations:  1)  a  minimum  assessment  battery  for  low-resoiuce 
applications  ,  and  2)  a  more  complete  battery  of  OWL  techniques  for 
circumstances  where  more  resources  are  available. 


3.7  Conclusions  and  Recommendations 
Several  conclusions  can  be  drawn  from  the  discussions  that  were  held: 


•  Within  the  Army  community,  there  is  real  concern  about  OWL  but  there 
seems  to  be  lack  of  conformity  in  how  to  address  it.  Everyone  seems  to 
have  their  own  way  of  doing  it  or  ignoring  it 

•  There  was  skepticism  of  any  project  as  wide-reaching  (i.e.,  going 
across  organizational  boundaries)  as  this  one. 

•  It  is  important  for  us  to  keep  abreast  of  other  Army  initiatives 
concerning  operator  workload. 

•  Any  program  or  methodology  developed  has  to  be  sensitive  to  resource 
limitations. 

•  The  methodology  must  be  able  to  accommodate  the  ASAP,  NDI,  and 
other  acquisition  strategy  procedures. 


3.8  Follow-on  Interview  Plans 


Those  individuals  in  Army  organizations  with  whom  we  spoke  provided  useful 
information.  We  will  continue  our  discussions  with  them  throughout  the  project  as 
appropriate  for  later  follow-ups.  Current  plans  are  to  keep  them  apprised  of  the  project 
progress  and  to  consult  with  them  again  in  the  future  concerning  the  handbooks.  Since 
our  primary  goal  is  to  provide  useful  information  in  the  most  appropriate  format,  the  Army 
community  of  users  will  be  consulted  regarding  the  development  of  the  handbooks. 
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4.  OUTLINE  OF  FINAL  PRODUCTS 


4  1  INTRODUCTION 

In  the  original  statement  of  work  (SOW),  five  major  areas  were  identified  to  be 
addressed  by  a  set  of  handbooks  which  Army  personnel  would  use  for  making  decisions 
on  OWL  during  the  materiel  acquisition  process  (MAP).  These  areas  were: 

•  How  and  where  to  use  handbooks 

•  Guidelines  for  how  to  estimate  OWL,  including  how  to  select  the  most 
appropriate  measures  of  OWL  for  a  given  new  system  design 

•  Guidelines  for  evaluating  OWL  during  concept  and  evaluation,  and  . 
developmental  and  operational  testing 

•  Alternative  methods  for  reducing  excessive  OWL  in  Army  systems,  and 

•  Recommendations  for  selecting  among  or  prioritizing  those  alternative 
OWL  reduction  methods. 

These  five  major  areas  were  originally  proposed  to  be  covered  by  three  related 
products:  an  overview  pamphlet  for  the  TRADOC  community  on  OWL,  and  two 
handbooks  addressing  OWL  techniques,  one  to  be  used  during  the  early  phases  of  the 
MAP  (pre-Milestone  1  activities)  and  the  other  to  be  used  during  system  development 
phases  of  the  MAP  (post-Milestone  1  activities). 

To  ensure  these  products  are  successfully  received  by  the  Army  community,  we 
developed  draft  outlines  depicting  each  product's  content  as  well  as  descriptions  of  the 
rationale  for  each  section  covered.  These  outlines  were  used  to  elicit  Army  personnel 
comments  and  reactions  in  order  to  ascertain  their  specific  needs  concerning  OWL  and  to 
"shape"  the  final  handbook  products.  It  was  the  first  step  in  our  ongoing  and  iterative 
development  of  user-oriented  products.  Examples  of  the  draft  outlines  that  were  discussed 
in  the  interviews  with  Army  personnel  may  be  found  in  Appendices  B  through  D. 

Based  on  meetings  with  Army  personnel,  we  identified  two  major  concerns  that 
needed  to  be  addressed  in  all  the  OWL  products  to  ensure  their  successful  use  and 
incorporation  into  the  work  activities  of  the  intended  users  of  these  products. 

•  A  thorough  description  of  what  operator  workload  is,  the  importance  of 
OWL  as  a  significant  determinant  of  system  performance,  and  how  it 
relates  to  system  requirements  and  design  issues. 
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•  A  descriptive  framework  to  show  the  integration  between  existing  Army 
requirements  (e.g.,  MANPRINT  requirements)  and  other  Army 
programs  (e.g.,  HARDMAN)  and  the  approach  prescribed  in  the 
handbooks.  How  the  OWL  program  complements  and  supports  these 
-  existing  Army  programs. 

Besides  these  overall  considerations,  each  outline  elicited  specific  comments  and 
suggestions.  These  user  reactions  will  be  discussed  in  the  respective  subsections  for 
each  proposed  product. 


4.2  TRADOC  PAMPHLET 


4.2.1  Original  Concept  in  Statement  of  Work  (SOW) 

The  TRADOC  overview  pamphlet  was  originally  conceived  as  a  guide  for 
TRADOC  personnel  "to  incorporate  workload  in  the  ROC  (Required  Operational 
Capability)".  This  pamphlet  was  seen  as  an  overview  that  described  what  is  meant  by 
OWL  and  how  to  address  it  in  a  ROC  .  It  would  highlight  the  issues  of  OWL  such  that 
TRADOC  managers,  (i.e.,  combat  developers),  could  recognize  the  need  to  address  OWL 
in  ROC  documents  and  assist  them  to  make  provisions  (e.g.,  requirements)  for  its  adequate 
assessments  in  subsequent  phases  of  the  MAP. 

4.2.2  Rationale  for  Ori ginal  Draft  Outline  and  Revision 

Our  original  draft  outline  expanded  the  scope  of  the  pamphlet.  We  envisioned  the 
pamphlet  providing  an  overview  emphasizing  the  need  to  address  OWL  throughout  the 
MAP  such  that  managers  in  TRADOC  and  AMC  who  are  in  positions  to  oversee  the 
development  of  systems  would  have  the  proper  framework  to  address  OWL  throughout  the 
cycle.  Otherwise,  issues  that  relate  to  OWL  could  be  overlooked  as  the  MAP  progresses. 
Such  an  emphasis  on  OWL  throughout  the  MAP  would  provide  the  correct  orientation  for 
OWL  to  be  addressed  in  a  proactive  mode  as  opposed  to  a  reactive  mode  of  fixing  past 
mistakes  attributed  to  excessive  operator  workload.  The  original  pamphlet  outline  dated 
9Feb87  is  found  in  Appendix  B. 

Based  on  our  discussions  with  Army  personnel  who  are  involved  with  various 
phases  of  the  MAP,  it  became  apparent  that  the  pamphlet  should  go  beyond  our  original 
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conceptualization  with  respect  to  our  intended  audience.  That  is,  there  seemed  to  be  a  lack 
of  awareness  and/or  concern  that  OWL  is  an  important  issue  to  be  addressed  throughout  the 
MAP. 

As  a  result,  the  pamphlet  needs  to  address  OWL  such  that  all  integral  players  in  the 
MAP  (e.g.,  TRADOC,  AMC,  OTEA,  AMSAA,  and  TECOM)  are  aware  of  the  importance 
of  addressing  OWL  throughout  the  MAP  since  all  can  contribute  to  preventing  OWL 
problems.  To  do  so  required  revising  our  initial  pamphlet  outline  such  that  ALL  key 
personnel  in  the  MAP  would  be  oriented  to  conceptualizing,  developing  and  evaluating 
systems  with  OWL  as  a  major  consideration.  This  is  especially  true  today  since  new 
technologies  (automation  via  software  innovations)  and  projected  manpower  reductions  are 
placing  a  potentially  heavier  burden  on  operators  to  mentally  perform  new  operations  that 
can  directly  impact  system  performance.  We  have  revised  the  manager's  pamphlet  outline 
to  reflect  this  orientation  toward  OWL  so  ALL  Army  personnel  involved  in  system 
development  realize  the  coordinated  effort  needed  across  organizations  to  ensure  that  OWL 
is  addressed  in  a  proactive  mode.  The  revised  pamphlet  outline  dated  23May87  follows 
this  subsection. 
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DATE:  23MAY87 


REVISED  MANAGER'S  OPERATOR  WORKLOAD  ASSESSMENT  PAMPHLET 


OUTLINE 

USER  PROFILE:  The  intended  users  for  the  pamphlet  are  the  managers  who  are  involved 
in  both  delineating  the  needs,  and  developing  the  requirements  for  a  new  system  as  well  as 
those  involved  in  evaluating  system  performance  in  which  the  soldier-in-the-loop  is 
considered  an  integral  part  of  the  evaluation.  This  user  is  not  interested  in  the  details  of 
workload  estimation  or  evaluation.  What  IS  of  interest  to  these  managers  is  high-level 
guidance  on  what  are  the  Army  requirements  regarding  workload,  and  what  high-level 
provisions  should  be  built  into  the  system  acquisition  strategy  for  the  assessment  of  OWL. 
The  orientation  of  this  pamphlet  is  to  instill  the  concept  of  operator  workload  (OWL)  in  the 
everyday  vocabulary  of  managers  such  that  it  is  addressed  in  a  proactive  mode  throughout 
the  MAP.  This  can  only  happened  if  ALL  managers,  irrespective  of  organizations,  are 
attuned  to  the  importance  of  OWL  as  a  major  factor  contributing  to  overall  system 
performance.  Each  manager  has  a  role  in  ensuring  that  OWL  does  not  adversely  affect 
overall  system  performance. 

FORMAT:  This  Pamphlet  will  be  structured  to  provide  a  concise,  easily  understood 
presentation  of  the  role  of  OWL  control  in  the  materiel  acquisition  process  (MAP).  Tables, 
charts,  flow  diagrams,  and  specific  examples  will  be  used  liberally  to  promote  quick 
apprehension  of  concepts. 

GOAL:  Provide  the  reader  with  an  overview  of  the  role  of  OWL  control  in  the  materiel 
acquisition  process,  including  the  nature  of  the  problem,  DoD/DA  documents  and 
requirements  concerning  OWL  control,  and  available  technologies  to  assist  ALL  Army 
managers  (e.g.,  TRADOC,  AMC,  OTEA,  AMSAA,  and  TECOM)  in  ensuring  OWL 
control.  Provide  guidance  in  accessing  other  OWL  control  resources,  especially  the  OWL 
Prediction  and  Evaluation  Handbooks. 
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LENGTH:  approximately  40-50  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B .  Traditional  factors  attributed  to  OWL,  e.g.,  physical  involvement  of  operators 

C.  New  technologies  affecting  system  concepts,  e.g.,  automation  via  software 

D.  New  factors  attributed  to  OWL,  e.g.,  mental/cognitive  involvement  of  operators 

E.  Impact  of  OWL  on  Army  Mission  Functions 

F.  Army  requirements,  specifications,  standards  and  regulations  for  OWL 

G.  Relationship  of  OWL  to  MANPRINT  Program 

H.  Contribution  of  ALL  managers  involved  with  the  MAP  in  addressing  OWL 

I.  Description  of  the  contents  of  this  pamphlet,  how  to  use  this  pamphlet 

STRATEGY:  Introduce  managers  to  the  key  OWL  concepts  and  regulations.  Provide  the 
proper  framework  on  how  to  use  this  handbook. 


II.  OVERVIEW  OF  OWL  FUNDAMENTALS 

A.  OWL  Performance  Model  -  factors  contributing  to  workload. 

B .  OWL  with  various  types  of  actions/behaviors  that  are  mental/cognitive  in  nature 

1 .  Searching  for  and  receiving  information 

2.  Identifying  objects,  actions,  events 

3.  Problem  solving 

4.  Decision  making 
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C.  OWL  considerations  for  system  types  with  respect  to  Mission  Areas 

D.  OWL  considerations  during  the  Materiel  Acquisition  Process  (MAP) 

1 .  Accelerated  Systems  Acquisition  Process  (ASAP) 

2.  Proactive  mode  vs.  reactive  mode  in  addressing  OWL 

3.  Key  questions  concerning  OWL  to  be  addressed  throughout  the  MAP 

E.  OWL  Assessment  Program 

1.  Prediction  (analytic  approach) 

2.  Evaluation  (empirical  approach) 

3.  Analysis  of  results  -  addressing  key  OWL  questions  as  revealed  by 
analysis  of  one's  results/data 

F.  OWL  Control  Plan 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  key  areas  and  steps 
involved  in  OWL  prediction/evaluation  and  its  relationship  to  the  materiel  acquisition 
process. 


III.  OWL  IN  REQUIREMENTS  ANALYSIS/CONCEPT  FORMULATION 

A.  TRADOC  perspective  -  combat  developers 

B .  AMC  perspective  -  program  managers 

C.  Evaluators/testers  perspective 

D.  OWL  trade  offs  in  concept  formulation 

E.  Development  of  a  preliminary  OWL  Control  Plan 

F.  Key  OWL  resources 

1 .  Documents 

2.  Organizations  (e.g,  HEL,  ARI) 

3.  Individuals  (e.g.,  HFE  specialists) 

G  The  TRADOC  Manager's  OWL  Concept  Formulation  Check  List 
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1.  What  should  the  TRADOC  combat  developer  be  ensuring  is 
accomplished. 

H.  The  AMC  Manager's  OWL  Concept  Formulation  Check  List 

1.  What  should  the  AMC  program  manager  be  ensuring  is  accomplished. 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  requirements  analysis  and  concept  formulation.  Provide  the 
manager  the  knowledge  to  integrate  the  OWL  Control  Plan  with  existing  requirements, 
programs,  and  other  control  plans  that  are  part  of  the  materiel  acquisition  process. 


IV.  OWL  IN  SYSTEM  DEVELOPMENT 

A.  TRADOC  perspective  -  system  managers 

B .  AMC  perspective  -  program  managers 

C.  Evaluators/testers  perspective 

D.  The  OWL  Control  Plan 

E.  Methods  for  assessment 

1 .  OWL  Prediction  (analytic  approach) 

2.  OWL  Evaluation  (empirical  approach) 

F.  Key  OWL  resources 

1.  Documents 

2.  Organizations  (e.g.,  HEL,  ARI) 

3.  Individuals  (e.g.,  HFE  specialists) 

G.  TRADOC  system  manager's  OWL  Check  List  for  OWL  Prediction 

H.  TRADOC  system  manager's  OWL  Check  List  for  OWL  Evaluation 

I.  AMC  program  manager's  OWL  Check  List  for  OWL  Prediction 

J.  AMC  program  manager's  OWL  Check  List  for  OWL  Evaluation 
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K.  Testers/Evaluators  Check  List  for  OWL  Prediction 

L.  Testers/Evaluators  Check  List  for  OWL  Evaluation 

STRATEGY:  Provide  the  manager  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  Assessment  Program  with  existing  requirements,  programs,  and  other 
control  plans  that  are  part  of  the  materiel  acquisition  process. 


V.  ITERATIVE  NATURE  OF  OWL  ASSESSMENT 

A.  Materiel  acquisition  process 

B .  System  design  decisions 

C.  Evolution  of  OWL  considerations 

STRATEGY:  Establish  the  concept  that  the  OWL  Assessment  Program  and  its  management 
and  control  are  evolving  processes  which  are  modified  as  the  materiel  acquisition  process 
progresses  and  requires  the  coordination  and  cooperation  of  all  Army  agencies  involved  in 
the  MAP. 


VI.  OWL  CONCERNS  AND  ARMY  SYSTEM  DEVELOPMENT  ITEMS 

A.  Non-Developmental  Items  (NDI) 

B.  Product  Improvements  (P3I,  PIP) 

STRATEGY:  Elaborate  on  the  special  circumstances  these  areas  present  for  addressing 
OWL  and  emphasis  the  need  to  ensure  that  OWL  does  present  itself  as  a  problem. 
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VII.  EXAMPLE 


A.  An  example  will  be  provided  that  delineates  the  various  managerial  responsibilities 
across  organizations  that  play  a  role  in  controlling  OWL  during  the  MAP  -  the 
development  and  implementation  of  their  respective  OWL  Control  Plans. 

STRATEGY:  Provide  the  user  with  a  concrete  example  of  an  overall  OWL  Control  Plan. 


VII.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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4.3  PREDICTION  HANDBOOK 


4,3.1  Ori^nal  Concept  in  Statement  of  Work  ('SOW') 

The  Workload  Prediction  Handbook  was  originally  conceived  as  a  guide  for 
addressing  OWL  during  the  new  system  concept  development  phase  of  the  MAP  (pre- 
Milestone  I  activities).  The  handbook  would  direct  Army  personnel  in  employing  OWL 
predictive  techniques  by  identifying  the  most  appropriate  predictive  technique  with  respect 
to  their  particular  needs  and  resources.  The  techniques  offered  would  lend  themselves  to 
identifying  OWL  issues  that  need  to  explore  and/or  address  in  tiie  further  refinement  of  the 
system  concept.  Use  of  such  techniques  would  give  valuable  direction  early-on  in  the 
MAP  so  to  minimize  potential  OWL  problems  in  later  phases  of  the  MAP. 


4.3.2  Rationale  for  Ori  ginal  Draft  Outline  and  Revision 


In  general,  our  original  draft  outline  attempted  to  provide  a  methodology  for 
identifying  the  most  appropriate  predictive  technique  for  a  given  system  concept.  It  also 
highlighted  the  importance  of  OWL,  its  relationship  to  system  performance  and  OWL 
evaluations  conducted  during  later  phases  of  the  MAP.  The  original  pamphlet  outline 
dated  9Feb87  is  to  be  found  in  Appendix  C. 

Based  on  our  meetings  with  Army  personnel,  we  identified  sections  in  the  outline 
that  needed  further  clarification  as  well  as  new  material  to  be  included  in  the  handbook. 
The  predictive  handbook  needs  to  address  and/or  clarify  the  following: 

•  Greater  emphasis  on  the  significance  of  addressing  OWL  early-on  in  the 
MAP  -  greater  likelihood  of  success,  (i.e.,  incorporating  OWL 
considerations  in  later  design  phases),  as  well  as  the  most  cost-effective 
point  in  time  to  offer  suggestions  for  design  changes  (i.e.,  least  expense 
involved  in  comparison  to  later  phases  of  the  MAP). 

•  The  complementary  relationship  between  the  use  of  this  handbook  and 
the  Army's  new  directive  to  address  soldier  issues  early-on  in  the  MAP 
-  MANPRINT  Program. 

•  The  relationship  between  the  guidance  offered  in  this  handbook  and  the 
methodologies  currently  used  by  the  Army  during  the  early  portions  of 
the  MAP,  (e.g.,  HARDMAN  and  EGA). 

•  How  the  results  generated  from  the  use  of  these  predictive  techniques 
can  drive  the  later  phases  of  the  MAP  in  controlling  OWL.  What  are  the 
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means  available  to  ensure  this,  (i.e.,  ROC  ,  future  test  plans,  DIDs, 
etc.). 

We  have  revised  the  predictive  handbook  outline  to  reflect  these  changes.  The 
revised  predictive  handbook  outline  dated  23May87  follows  this  subsection. 
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DATE:  23MAY87 


REVISED  WORKLOAD  PREDICTION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Prediction  Handbook  is  intended  for  Army  personnel 
(e.g.,  TRADOC  combat  developer  and  system  manager)  during  the  concept  and  'Carly 
design  phases  of  the  materiel  acquisition  process  (MAP).  This  user  is  interested  in  the 
different  OWL  measures  and  techniques  applicable  during  early  design.  This  user  is 
typically  the  person  who  (1)  makes  the  decision  of  which  OWL  assessment  tools  to  use, 
and  (2)  adapts  those  tools  to  fit  the  specific  needs  and  characteristics  for  the  system  of 
interest.  To  guide  the  user  in  performing  these  functions,  the  handbook  will  identify 
(based  on  system  requirements  and  specific  design  Objectives)  the  workload  assessment 
methodology  needed  via  a  "matching  model"  procedure  such  that  an  optimal  OWL 
Assessment  Battery  is  offered.  The  handbook  provides  guidance  on  the  identification  and 
implementation  of  the  OWL  Assessment  Battery  for  all  types  of  systems. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
user  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an  OWL 
assessment  program  during  early  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  assessment  program  with  existing  requirements,  programs  (e.g., 
MANPRINT) ,  and  methodologies  (e.g.,  EGA  and  HARDMAN)  that  are  used  in  the  early 
phases  of  the  materiel  acquisition  process  (MAP). 

LENGTH:  approximately  75-100  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B .  Traditional  factors  attributed  to  OWL,  e.g.,  physical  involvement  of  operators 

C.  New  technologies  and  factors  affecting  system  concepts,  (e.g.,  automation  via 
software) 

D.  New  factors  attributed  to  OWL,  (e.g.,  mental/cognitive  involvement  of  operators) 

E.  Impact  of  OWL  on  overall  system  performance 

F.  Army  requirements,  specifications,  standards,  and  regulations  for  OWL 

G.  Relationship  of  OWL  to  MANPRINT  Program 

H.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  operator 
workload  assessment  program  via  OWL  predictive  techniques  during  concept  and 
preliminary  design  phases 

I.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  key  OWL  concepts  and  requirements.  Orient  the  user  to 
the  critical  concept  -  OWL  predictive  assessment  techniques  are  critical  for  identifying 
OWL  issues  such  that  solutions  concerning  OWL  (overload)  can  be  reached  in  a  cost- 
effective  way  (proactive  mode).  Otherwise,  potential  OWL  problems  will  probably  be 
addressed  in  a  reactive  mode,  (i.e.,  fixing  past  mistakes).  Provide  the  framework  on  how 
to  use  the  Handbook. 
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11.  OVERVIEW  OF  OPERATOR  WORKLOAD  PREDICTION 


A.  What  is  OWL  prediction? 

B .  Relationship  between  methodology  offered  in  this  document  and  existing  Army 
methodologies,  (i.e.,  EGA,  HARDMAN,  Task  Analysis) 

C.  The  relationship  between  OWL  prediction  &  OWL  evaluation 

D.  Factors  to  consider  for  OWL  prediction 

1.  System  requirements 

2.  Operator  capabilities/skills/behaviors  required 

3.  OWL  performance  model  -  performance  factors  to  consider 

4.  OWL  assessment  techniques 

E.  Methodology;  Matching  model  for  establishing  an  OWL  Predictive  Assessment 
Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
prediction  (analytic  approach)  and  its  relationship  to  OWL  evaluation  (empirical  approach) 


III.  OPERATOR  WORKLOAD  PREDICTION  METHODOLOGY 

A.  Determine  system  performance  requirements 
1 .  Potential  Sources: 

a.  O  &  O  Plans  (Operational  &  Organizational) 

b.  JMSNS  (Justification  of  Major  Systems  New  Starts) 

c.  ECA  (Early  Comparability  Analysis) 

B .  Determine  operator  actions/behaviors  for  system  usage 
1 .  Sources 

a.  Requirement  documents  (O  &  O  Plans,  ROCs) 

b.  Task  analyses 

c.  Expert  opinions  (SMEs) 
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d.  Comparative  systems  (e.g.,  SMMP,  EGA) 

e.  Mock-ups 

f.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system  usage 
(cf.,  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell,  & 
Shearer,  1964) 

-  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 

c.  Problem  solving 

d.  Decision  making 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
methodology  (decision  making  process) 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  predictive  assessment  battery 

2.  Decision  tables  for  OWL  methodologies  that  are  applicable 

F.  Analytic  assessment  methods 

1 .  Purpose:  Prediction  of  OWL  &  "chokepoints” 

2.  Sensitivity 

3.  Representation  of  OWL  issues/behaviors 

4.  Techniques 

a.  Expert  opinions 

b.  Comparisons 

c.  Simulations/Models 

d.  Math  models 

e.  Task  analytic  methods 

5.  Interpretation  of  assessment  results 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an 
OWL  Predictive  Assessment  Program.  Provide  the  user  the  know-how  to  integrate  the 
OWL  Assessment  Program  with  existing  requirements,  programs,  and  methodologies  that 
are  part  of  the  materiel  acquisition  process. 
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IV.  ITERATIVE  NATURE  OF  OWL  PREDICTION 


A.  Materiel  acquisition  process 

B.  System  design  decisions 

C.  How  to  address  OWL  issues  in  the  MAP,  (i.e.,  ROC,  SMMP,  and  Test  Plans) 

STRATEGY:  Establish  the  concept  that  the  OWL  assessment  program  (developed  from 
using  this  Handbook)  is  an  evolving  program  which  will  be  modified  as  the  materiel 
acquisition  process  progresses.  OWL  must  be  monitored  and  controlled  by  establishing 
requirements,  (i.e..  Measures  of  Effectiveness  [MOEs]  in  the  ROC),  that  ensure  activities 
conducted  after  Milestone  I  address  OWL  issues  that  were  identified  from  the  use  of  the 
OWL  predictive  techniques. 


V.  EXAMPLES 

A.  Examples  will  be  provided  for  each  of  the  assessment  techniques. 

STRATEGY:  Provide  the  user  with  concrete  examples  of  OWL  predictive  assessment 
programs.  Provide  reality  to  the  OWL  prediction  methodology  described  in  the  Handbook. 


VI.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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4.4  EVALUATTON  HANDBOOK 


4.4. 1  Original  Concept  in  Statement  of  Work  (SOW) 

The  Workload  Evaluation  Handbook  was  originally  conceived  as  a  guide 
for  addressing  OWL  during  the  conduct  of  concept  evaluation,  and  developmental  and 
operational  testing  (post-Milestone  I  activities).  The  handbook  would  direct  Army 
personnel  in  employing  OWL  evaluation  techniques  (subjective,  physiological,  and 
objective/task  measures)  by  identifying  the  most  appropriate  empirical  technique  with 
respect  to  their  particular  needs  and  resources.  The  techniques  offered  would  lend 
themselves  to  be  incorporated  in  any  system  evaluation  effort  and  would  complement  any 
existing  data/information  on  OWL  collected  earlier  in  the  MAP  (predictive  techniques). 
These  techniques  provide  data  to  substantiate  design  solutions/decisions  to  minimize  OWL 
as  well  as  direct  future  refinements  for  system  design. 


4.4.2  Rationale  for  Original  Draft  Outline  and  Revision 


In  general,  our  original  draft  outline  attempted  to  provide  a  methodology 
for  identifying  the  most  appropriate  evaluative  technique  for  a  given  system  during  the 
various  developmental  phases  phases  after  Milestone  I  activities.  It  also  highlighted  the 
importance  of  OWL,  its  relationship  to  system  performance  and  previous  OWL 
assessments  conducted  earlier  during  the  MAP  (OWL  predictive  techniques).  The  original 
handbook  outline  dated  9Feb87  is  to  be  found  in  Appendix  D. 

Based  on  our  meetings  with  Army  personnel,  we  identified  Army  concerns  that 
need  to  be  addressed  in  the  Evaluation  Handbook  These  concerns  are  the  following: 

•  Emphasis  on  the  coordination  of  previous  OWL  assessments  (OWL 
predictive  techniques)  with  subsequent  testing  and  evaluation  such  that 
provisions  are  established  (e.g.,  ROC,  SMMP)  to  ensure  that  OWL 
concerns  already  identified  are  addressed  adequately  in  the  later  phases 
of  the  MAP. 

•  Elaboration  on  the  key  roles  that  the  various  testing  and  evaluation 
agencies  (OTEA,  AMSAA,  and  TECOM)  as  well  as  system  and 
training  agencies  (AMC  and  TRADOC)  play  in  addressing  OWL  issues. 

All  have  a  contribution  to  make  in  controlling  OWL. 

•  Address  the  applicability  of  the  methodology  proposed  in  the  handbook 
with  respect  to  the  Army's  accelerated  programs  for  system  acquisition. 
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(i.e.,  ASAP  and  NDI),  and  the  evolutionary  development  of  systems, 

(i.e.,  PIP  and  P3I). 

We  have  revised  the  evaluation  handbook  to  reflect  these  concerns.  The  revised 
evaluation  handbook  outline  dated  23May87  follows  this  subsection. 
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DATE:  23MAY87 


REVISED  WORKLOAD  EVALUATION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Evaluation  Handbook  is  intended  primarily  for  the 
Army  community  involved  with  developmental  testing  and  evaluation  (DT&E).  These 
users  are  interested  in  interpreting  the  results  of  the  various  workload  assessments 
conducted  throughout  the  development  cycle  and  are  primarily  concerned  with  system 
evaluation  from  many  different  perspectives  (e.g.,  TRADOC,  AMC  ,  OTEA,  AMSAA, 
and  TECOM).  These  evaluators  and  users  of  data  from  test  and  evaluation  (T&E)  are  also 
interested  in  how  to  perform  OWL  analysis.  In  addition,  they  are  concerned  with  the  actual 
constraints  placed  by  real-world  implementation  of  a  workload  assessment.  Such 
constraints  involve  traditional  (and  non-traditional)  limits  on  testbed  resources  (e.g., 
subjects,  time,  and  funding).  The  T&E  users  are  also  concerned  with  how  to  transform 
OWL  data  and  information  into  recommendations  for  system  design. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
reader  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing 
an  OWL  assessment  program.  Provide  the  user  the  know-how  to  integrate  the  OWL 
assessment  program  with  existing  requirements,  programs,  and  methodologies  that  are  part 
of  the  military  system  acquisition  cycle. 

LENGTH:  approximately  125-150  pages 
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CONTENTS 


1.  BsrmoDUcnoN 

A.  Definition  of  Operator  Workload  (OWL) 

B.  Traditional  factors  attributed  to  OWL,  (e.g.,  physical  involvement  of  operators) 

C.  New  technologies  and  factors  affecting  system  concepts,  (e.g.,  automation  via 
software) 

D.  New  factors  attributed  to  OWL,  (e.g.,  mental/cognitive  involvement  of  operators) 

E.  Impact  of  OWL  on  overall  system  perforaiance 

F.  Army  requirements,  specifications,  standards,  and  regulations  for  OWL 

G.  Relationship  of  OWL  to  MANPRINT  Program 

H.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  operator 
workload  assessment  program  via  OWL  evaluative  techniques  during  concept 
evaluation  and  developmental  and  operational  testing. 

I.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  the  key  OWL  concepts  and  requirements.  Orient  the 
user  to  the  critical  concept  -  OWL  Evaluative  Techniques  are  crucial  for  determining  the 
feasibility  of  design  decisions  as  they  relate  to  minimizing  OWL.  Provide  the  proper 
framework  on  how  to  use  the  handbook. 


II.  OVERVIEW  OF  OPERATOR  WORKLOAD  EVALUATION 


A.  What  is  OWL  evaluation? 


r  A  A 
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Who  is  involved?  (TRADOC,  AMC,  OTEA,  AMSAA,  TECOM, ) 


B. 

B .  The  relationship  between  OWL  evaluation  &  OWL  prediction 

C.  Factors  to  consider  for  OWL  evaluation 

1 .  System  requirements 

2.  OWL  assessment  program  constraints 

a.  Subjects 

b.  Time 

c.  Funding 

d.  Etc. 

3.  Operator  capabilities/skillsAjehaviors  required 

4.  Earlier  OWL  assessments  (OWL  prediction) 

5 .  OWL  performance  model  -  performance  factors  to  consider 

6 .  OWL  empirical  assessment  techniques 

D.  Methodology:  Matching  model  for  establishing  an  OWL  Evaluative  Assessment 
Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
evaluation  (empirical  approach)  and  its  relationship  to  OWL  prediction  (analytic  approach), 
and  system  design. 


III.  OWL  ASSESSMENT  (TEST  AND  EVALUATION) 

A.  Determine  system  performance  requirements 
1,  Sources 

a.  Requirement  documents  (ROC,  and  O  &  O  Plans) 

B .  Determine  operator  actions/behaviors  for  system  usage 

1.  Sources 

a.  Task  analyses 

b.  Expert  opinions 

c.  Comparative  systems 

d.  Mock-ups 
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e.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system  usage 
(cf.,  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell,  & 
Shearer,  1964)  -  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 

c.  Problem  solving 

d.  Decisionmaking 

C.  Determine  stage  in  materiel  acquisition  process 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
(decision  making  process) 

1 .  For  example,  environmental  factors  such  as  noise,  vibration,  heat,  and 
cold 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  Assessment  battery 

2.  Decision  tables  for  workload  methodologies  that  are  apphcable 

F.  Estabhsh  framework  for  the  evaluation 
1.  Task  scenarios 

G .  Empirical  Assessment  Methods 

1 .  Purpose:  Evaluate  design  decisions 

2.  Sensitivity 

3.  Task  scenarios:  representation 

4.  Techniques 

a.  Operator  opinion 

b.  Primary  task 

c.  Secondary  task 

d.  Physiological  responses 

5.  Interpretation  of  assessment  results  to  impact  system  design 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  implementing  an  OWL  evaluative 
assessment  program  -  test  and  evaluation.  Provide  the  user  the  know-how  to  integrate  the 
OWL  assessment  program  with  existing  requirements,  programs,  and  methodologies  that 
are  part  of  the  materiel  acquisition  process. 
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IV.  nERATTVE  NATURE  OF  OWL  EVALUATION 


A.  Materiel  acquisition  process  (MAP)  and  Accelerated  acquisition  process  (ASAP) 

B.  System  design  decisions 

C.  Army  agencies  involved  in  the  iterative  nature  of  OWL  assessment  -  (TRADOC  and 
AMC  as  key  players ) 

STRATEGY:  Establish  the  concept  that  OWL  assessment  (test  and  evaluation)  is  an 
iterative  process  that  is  conducted  throughout  the  materiel  acquisition  process  to  ensure 
OWL  is  not  a  system  problem. 


V.  OWL  CONCERNS  AND  ARMY  SYSTEM  DEVELOPMENT  HEMS 

A.  Non-Developmental  Items  (NDI) 

B.  Product  Improvements  (e.g.,  P3I,  PIP) 

STRATEGY:  Elaborate  on  the  special  circumstances  these  areas  present  for  addressing 
OWL  and  show  the  applicability  of  the  methodology  represented  in  this  handbook  to  these 
set  of  circumstances. 


VI.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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5.  SELECTION  OF  REPRESENTATIVE  SYSTEMS 


5.1  Introducrion 

The  three  representative  system  selected  during  the  present  effort  are  crucial  to  later 
efforts.  They  provide  the  test-bed  upon  which  the  validation  of  OWL  measures  and 
methodology  will  be  later  conducted  (Task  4).  In  addition,  these  also  provide  the 
framework  for  illustration  of  applications  which  will  augment  of  the  documentation  of  our 
methods  (OWL  Handbooks)  which  will  be  subsequently  be  prepared  (Task  5).  Beyond 
thishowever,  it  is  intended  that  the  analyses  of  the  prototype  systems  will  provide 
significant  benefits  for  their  development  and  the  Army.  Delineated  in  the  following  are: 
the  selection  methodology  (5.2),  description  of  the  selected  systems  (5.3),  and  brief 
discussion  of  on-going  efforts  (5.4). 


5.2  Methodology 

A  two-stage  approach  was  applied  for  selection  of  representative  systems  based 
upon  Interview  Inputs  and  Army  Sponsor  Guidance. 

5.2.1  Interview  Inputs 

The  first  stage  involved  identification  of  candidate  systems  during  our  interviews  of 
potential  users  and  cognizant  personnel  within  the  Army  (cf.,  Sect.3.2).  Typically 
subsequent  to  description  of  the  OWL  Assessment  Program,  interviewees  were  presented 
with  candidate  system  selection  criteria.  Most  salient  of  these  were  requirements  to:  (1) 
select  candidates  requiring  the  broad  range  of  predictive  and  evaluative  OWL  techniques 
and  (2)  select  systems  which  could  be  realistically  impacted  within  the  time  frame  of  the 
program  efforts.  The  first  of  these,  it  is  noteworthy,  was  reflected  in  interest  with  systems 
in  different  phases  of  the  materiel  acquisition  process,  and  under  different  combat 
developers  consequently  representing  a  range  of  systems.The  second  requirement  pointed 
toward  systems  undering  rapid  (NDI)  or  near-term  changes  in  status.  The  interviews 
proved  fruitful  and  pointed  toward  more  than  a  dozen  individual  or  families  (e.g.,  AFV)  of 
systems  for  joint  consideration  with  ART 
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5.2.2  Army  Sponsor  Guidance 


Candidates  systems  were  weighed  with  respective  to  theobjectives  and  schedule  of 
the  present  effort  in  a  joint  meeting  6-7  May  at  the  ART  Field  Unit,  Ft.  Bliss.  Based  upon 
information  obtained  in  interview  follow-ups,  system  candidates  were  first  screened  with 
respect  to  the  program  schedules  of  the  various  systems  as  well  as  assessability  by  the 
evaluation  team.  For  surviving  candidates,  operators  of  interest  were  identified  and 
evaluated.  Group  evaluations  were  made  with  respect  to  operator  gross  environmental 
exposure  (impact);  estimated  relative  levels  of  perceptual,  cognitive,  motor,  and 
communication  workload;  as  well  as  physiological  workload  levels  imposed  by  manual 
tasks  (lifting,  pushing,  carrying  etc.).  Representative  systems  were  then  selected  so  as  to 
insme  a  range  of  perceptual,  cognitive,  motor,  and  communication  workload  levels  across 
systems. 


5.3  Selected  Representative  Systems 

The  selected  representative  systems  include:  (1)  Line  of  Sight-Forward  (Heavy) 
[LOS-F(H)];  (2)  Automatic  Target  Handoff  System  [ATHS];  as  well  as  (3)  a  Remotely 
Piloted  Vehicle  (RPV[AQUILA  or  lEW-UAV(I)].  In  the  following,  each  of  these  systems 
will  initially  be  characterized  with  regard  to  nature,  type  of  acquisition  and  status,  as  well  as 
operators  of  interest.  The  comparison  of  the  character  of  anticipated  OWL  will 
subsequently  be  overviewed  across  systems  for  the  operators  of  interest. 

S.31  T.OS-FtFn 

LOS-F(H)  is  one  of  five  components  of  the  Forward  Area  Air  Defence  (FAAD) 
System.  FAAD  as  a  whole  is  designed  to  defend  force  and  critical  assets  against  rotary 
wing  and  fixed  wing  aircraft  threats  during  24  hour  day  and  night  operations,  a 
countermeasures  environment,  and  adverse  visibility  and  weather  conditions.  The  LOS- 
F(H)  component  of  FAAD  will  be  a  self  propelled,  armored,  highly  mobile,  platform  with 
a  primary  armament  of  launch-ready  missiles  as  well  as  a  complementary  weapon 
providing  full  coverage  of  air  defence  within  the  dead  zone  of  the  missile.  It  is  designed  to 
provide  front  line  air  defence  against  attacks  by  high  performance  ground  attack  aircraft, 
attack  helicopters,  as  well  as  self  defence  against  armored  vehicles  and  ground  targets. 
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LOS-F(H)  is  being  acquired  under  a  NDI  strategy  aimed  at  fielding  during  1991  which 
provides  for  evaluation  of  the  breadth  of  measures  of  OWL.  Candidate  selection. 
Milestone  I/II  of  the  ASAP,  is  scheduled  to  be  made  on  the  basis  of  a  competitive  shoot-off 
to  be  conducted  during  September  to  December  1987.  Follow-on  testing  of  the  selected 
candidate  as  will  as  Force  Development  Test  and  Evaluation  (FDTE)is  scheduled  for  Fiscal 
Year  (FY)  88.  LOS-F(H)  has  two  operators  of  particular  interest  with  respect  to  workload 
that  may  be  nominally  designated  across  system  candidates  as:  (1)  Gunner  and  (2)  Squad 
Leader. 

5.3.2  Automatic  Target  Handover  System  FATHSI 

The  ATHS  is  designed  to  provide  secure  jam  resistant  digital  communications  and 
is  envisioned  as  being  a  subsystem  in  a  variety  of  future  helicopter  and  other  vehicles. 
With  associated  Navigation  and  Interrogation  Friend  or  Foe  (IFF)  Subsystems,  it  is 
currently  scheduled  for  integration  into  the  APACHE  AH-64A  Aircraft  (the  draft  RFP  is 
under  review).  This  integration  is  for  facilitation  of  intelligent  automatic  digital  data 
processing  for  missiles,  weapons,  and  related  information  transfer  between,  other  AH- 
64A,  Scout  Aircraft,  and  ground  units  via  digital  waveforms  compatible  with  TACFIRE. 
For  the  AH-64A,  the  integration  is  to  allow  simultaneous  operation  by  dual  [Pilot  and 
Copilot/Gunner  (CPG)]  control  and  display  units  which  shall  be  night-vision  goggle 
compatible.  It  is  believed,  however,  that  in  flight  the  primary  operator  will  be  the  CPG 
because  of  the  flight  control  responsibilities  of  the  Pilot.  The  CPG  will  have  a  message 
received  indicator  "that  can  be  viewed/heard  during  all  workload  conditions".  The  CPG  is 
the  operator  which  has  been  identified  for  detailed  workload  evaluation. 

5.3.3  Anuila/  TEW-T TAVm  Remotely  Piloted  Vehicles  (RPVs) 

The  Aquila  and  Unmanned  Air  Vehicle  -  Intelligence  Electronic  Warfare  (Interim) 
[lEW-UAV(I)]  are  RPV  components  of  representative  systems  pointed  to  as  having 
growing  significance  for  the  Army.  Of  these,  the  Aquila  based  system  was  judged 
somewhat  more  suitable  for  the  goals  of  the  present  (OWL)  effort  because  of  its 
developmental  maturity  and  history.  However  as  it  is  under  Full  Scale  Engineering 
Development  (FSED),  with  an  impending  scheduled  ASARC  in  June  1987,  and  it  was 
deemed  prudent  to  retain  as  a  backup  the  lEW-UAV(I).  Currently  with  broad  opportunities 
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for  the  present  (OWL)  efforts  because  of  its  status  as  a  NDI,  the  IEW-UAV(I)  is  being 
selected  on  the  basis  of  an  on-going  flyoff  and  will  be  used  for  development  of  the 
specifications  of  final  UAVs  by  Ft.  Huechuca.  Based  upon  initial  analyses  and  the  status 
of  the  lEW-UAV(I),  it  was  judged  that  for  the  present  purposes  it  could  be  considered 
analogous  to  (a  larger)  Aquila  (with  similar  perceptual,  cognitive,  motor,  and 
communication  performance  and  workload  problems).  Consequently,  Aquila  will  be 
delineated  in  the  following  both  with  regard  to  itself  and  as  a  surrogate  for  the  lEW- 
UAV(I). 

The  Target  Acquisition/Designation  and  Aerial  Reconnaissance  System  (TADARS) 
Aquila  RPV  is  an  'eye  in  the  sky  system'  designed  to  provide  the  ground  commander 
realtime  battlefield  information  by  detection,  recognizing,  identifying  and  designating 
enemy  forces  (ART,  1987).  The  Acquila  itself  is  a  tailless  mid-wing  tactical  mini-plane 
with  a  rear-mounted  pusher  propeller  engine.  Included  as  part  of  the  system  are  a  stabilized 
TV  sensor  and  a  laser  rangefinder/designator  on  the  aircraft,  an  antijam  data  link  for 
communication  with  a  ground  control  station,  as  well  as  personnel  for  operation.  Basically 
fixed  during  operations,  the  ground  control  station  may  be  noted  as  being  located  in  an 
protected  shelter  in  the  rear  of  a  MS  14  5-Ton  Truck.  More  interestingly,  the  data  link  may 
be  noted  as  having  has  seven  (0-6)  levels  of  function  which  may  impact  on 
mission/modes(Search,  Artillery  and  Track)  performance  and  workload.  Although  more 
recent  flight  tests  have  implications  regarding  this  of  which  we  are  not  fully  apprised  (e.g., 
GAO,  1986;  ARI,  1987,  p.4),  Hershberger  et  al.,  1983  have  previously  reported 
suggestive  results  from  "man  in  the  loop"  simulations.  Aquila  has  three  operators  of  which 
two  have  been  identified  for  detailed  workload  evaluation: 

(1)  Vehicle  Operator  (VO) ,  and 

(2)  Mission  Commander  (MC). 

5.3.4  OWL  Overview  For  The  Selected  Systems 

Figure  5.3.4-1  summarizes  salient  features  of  operator  workload  developed  during 
the  group  evaluation  meeting  6-7  May  at  Ft.  Bliss  (cf.,  5.2.2).  Examining  this  table,  it 
may  be  seen  that  except  for  the  Gunner  in  the  LOS-F(H)  (during  loading  operations), 
physiological  workload  is  expected  to  relatively  constant  and  minimal  (Low).  The  group 
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judgements  of  perceptual,  cognitive,  motor,  and  communication  workload  levels  may  also 
be  seen  to  span  the  range  (Low-High). 


5.4  Discussion 

The  three  selected  representative  systems  are  the  test-beds  upon  which  the 
validation  of  OWL  measures  and  methodology  will  be  later  conducted  (Task  4).  Although 
a  large  step,  their  selection  is  only  the  beginning  of  the  development  of  an  overall  and 
system  specific  plans  for  the  methodological  validation  be  developed  during  the  next  phase 
of  efforts  (Task  3).  To  insure  the  success  of  the  efforts,  as  well  as  provide  for  the 
significant  benefit  for  the  representative  systems  through  OWL  analysis,  will  require  long 
and  close  collaboration  with  system  and  other  cognizant  personnel.  In  anticipation  of  this, 
personnel  were  identified  during  our  interviews  of  users  and  other  selected  personnel 
within  the  Army  (cf.,  Sect.3.2).  Cognizant  personnel  are  currently  being  contacted 
regarding  the  selection  of  systems  and  coordination  of  joint  efforts  in  beginning. 
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OWL  CATEGORY  LEVEL 


Figure  5.3.4- 1  OWL  Overview  for  the  selected  systems 


6.  OTHER  PROGRAM  PROGRESS 


Noteworthy  progress  has  been  toward  program  goals  which  are  tangential  to  the 
subject  of  the  present  report  (re:  Task  2).  OWL  information  collection  and  analysis  has 
been  and  is  underway  in  preparation  for  the  evaluation  of  workload  measures  which  is  to 
be  conducted  as  part  of  the  next  phase  of  effort  (i.e..  Task  3).  For  the  interested  reader, 
details  of  the  status,  nature,  and  methods  of  this  collection  may  be  found  delineated  in  the 
appendix  describing  the  OWL  Information  System.  The  OWL  information  analysis  effort, 
it  is  pertinent  to  observe,  is  based  upon  a  taxonomy  of  workload  assessment  methods  with 
two  broad  classes: 

•  Analytic  -  predictive  techniques  that  may  be  applied  earliest  in  system 
design  before  "man  in  the  loop"  studies;  and 

•  Empirical  -  operator  workload  assessments  that  are  taken  during 
simulator,  prototype,  or  system  evaluations. 

Analytic  techniques  have  initially  received  our  greatest  interest  and  attention.  This 
is  partially  because  (although  not  in  the  context  of  the  system  evaluation  objectives  of  the 
Army):  (1)  the  empirical  techniques  have  received  considerable  attention,  (e.g.,  O'Donnell 
and  Eggemeier,  1986);  and  (2)  a  model  matching  empirical  techniques  with  user 
requirements  has  been  been  reported  (Casper,  et  al.,1986)  although  apparently  currently 
unavailable.  Our  motivation  for  focusing  on  the  analytic  techniques  lies  in  their  application 
during  the  earliest  developmental  stage  where  the  greatest  design  flexibility  is  available  at 
the  least  cost  as  pointed  out  earlier  in  this  report  (cf..  Sect.  2).  Thus  far,  we  have  classified 
these  analytic  techniques  into  five  categories: 

•  Comparability  Analysis  -  This  category  involves  the  application  of 
existing  workload  data  from  an  comparable  earlier  system  to  estimate  the 
workload  for  the  system  under  development  (e.g.,  Shaffer,  et  al., 

1986). 

•  Mathematical  Models  --  Various  mathematical  techniques  have  been 
used  in  a  theoretical  context  for  a  long  time.  Transfer  functions, 
information  theory,  and  queuing  theory  are  some  of  the  techniques  of 
this  category  (e.g..  Senders,  1964).  These  provide  basic  limits  or 
boundaries  which  may  be  applied  during  "front  end  analysis"  with 
regard  to  OWL. 

•  Expert  Opinion  -  The  category  relies  on  the  opinion  of  experts  who 
have  intimate  working  knowledge  of  the  mission  objectives,  the 
operational  environment  and  the  workload  of  comparable  systems  . 

This  category,  in  contrast  to  listed  below,  relies  upon  experts  to  identify 
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choke-points  and  does  not  require  formal  task  analyses  (Zachary, 
1981). 

•  Task  Analytic  Techniques  -  This  category  involves  the  development  of 
a  mission  profile  which  represents  the  way  the  system  is  to  be  used.  A 
number  of  investigators  have  used  the  approach  (e.g.,  Stone,  Gulick, 
and  Gabriel,  1985).  The  profile  makes  it  possible  to  perform  a  task 
analysis  for  a  given  work  station  and  translate  it  into  activity  profiles  as 
a  function  of  time.  This  procedure  will  uncover  major  situations  in 
which  the  time  available  to  perform  the  task  (mission)  exceeds  the  time 
available. 

•  Simulation  Models  —  These  attempt  to  model  human  behavior  and 
thereby  predict  performance.  Some  operate  at  the  level  of  operator  tasks 
(task  analysis)  to  predict  level  of  performance  (e.g.  Siegel  and  Wolf, 
1969).  Others  extend  the  task  analysis  approach  and  carry  the  scenario 
into  far  more  detail;  macro  and  micro  models  of  components  of  the  task  . 
are  used  to  build  accuracy  and  time-line  projections  of  human 
performance  (e.g.,  Harris,  et  al.,  1986). 


We  are  presently  documenting  a  procedure  for  the  selection  from  among  these 
analytic  categories  based  on  user  requirements  and  resources.  As  an  initial  step,  this 
procedure  distinguishes  OWL  comparability  analyses  from  other  categories  by  the 
requirement  for  comparable  system  data.  In  subsequent  steps  going  from  Mathematical 
Models  to  Simulation  Models,  the  other  categories  are  distinguished  by  their  increasing 
requirements  for  formal  system  and  operator  task  definitions.  Our  plan  is  to  shortly 
complete  an  overview  of  our  procedure  for  selection  among  analytic  assessment  of  OWL 
(Hill,  et  al.,  in  preparation). 
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7.  FUTURE  PLANS 


7. 1  Future  Plans  for  Subsequent  Tasks 

Task  3  will  consist  of  a  critical  review  and  evaluation  of  OWL  measurement 
techniques  (predictive  and  evaluative)  and  the  development  of  validation  plans  for  the  OWL 
methodology  to  be  applied  to  the  three  Army  prototype  systems. 

Validation  plans  will  be  prepared  to  include  OWL  measures  that  offered  the  greatest 
utility  and  impact  on  the  prototype  systems  described  earlier  [LOS-F(H),  ATHS,  and 
Aquila/IBW-UAV(I)].  For  each  system  ,  a  goal  of  the  validation  plans  will  be  to  utilize 
families  of  OWL  techniques.  For  example,  a  family  of  predictive  techniques  (e.g.,  expert 
opinion,  simulation  models)  will  be  applied  to  systems  in  order  to  assess  their  relative 
utility  in  providing  similar  types  of  information  before  man-in-the-loop  simulations. 
Similarly,  a  family  of  evaluative  techniques  (e.g.,  subjective  techniques  such  as  SWAT, 
TLX,  and  Modified  Cooper-Harper  Scales)  will  be  applied  to  systems  in  order  to  assess 
their  relative  utility  in  providing  information  during  man-in-the-loop  evaluations.  The 
validation  plans  will  be  structured  to  be  reflective  of  crew  workload  issues  to  the  extent 
possible  within  the  validation  effects.  OWL  analyses  will  also  be  directed  at  ensuring 
benefit  for  the  selected  systems. 

In  Task  4,  the  plans  generated  in  Task  3  will  be  implemented.  We  will  conduct 
studies  to  validate  the  OWL  measures  and  methodology  while  performing  OWL  analyses 
on  the  three  selected  systems.  In  addition  to  demonstrating  and  validating  the  OWL 
methodology,  it  is  intended  that  the  analyses  of  the  prototype  systems  will  provide 
significant  and  direct  benefits  for  the  development  of  these  systems.  This  is  extremely 
important  since  the  success  of  this  OWL  program  will  be  direct  function  of  the  Army 
community's  reaction  to  our  approach.  The  more  we  can  document  our  "successes"  in 
using  this  methodology,  the  greater  the  likelihood  that  the  Army  community  will  be 
receptive  to  using  our  methodology  (products). 

Task  5  will  address  the  production  of  the  primary  documentation  of  the  OWL 
program.  We  will  produce  three  products:  Manager's  OWL  Pamphlet,  OWL  Prediction 
Handbook  and  an  OWL  Evaluation  Handbook.  A  Post-Contract  Survey  form  will  be 
prepared  to  provide  an  efficient  vehicle  to  assess  the  degree  to  which  these  OWL 
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documents  have  met  their  goals  for  the  Army.  The  technical  report  detailing  the  scientific 
basis  for  the  information  contained  in  the  pamphlet  and  handbooks,  and  discussing  further 
research  in  the  area  of  controlling  operator  workload  will  be  prepared  as  the  final  product 
for  the  Army. 
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Appendix  A 


OPERATOR  WORKLOAD  (OWL)  SURVEY 
ARI  CONTRACT  NO.  MDA903-86-C-0384 


PLEASE  RETURN  TO: 
ANALYTICS  INC.  (  ATTN;  OWL) 
2500  MARYLAND  ROAD 
WILLOW  GROVE,  PA.  19090 
(215)  657-4100  x.164 


PG.2 


RFSPONSIBII-ITIES/RQLES  IN  THE  MATERIEL  ACQUISITION  PROCESS  (MAP) 

1 .  PLEASE  INDICATE  YOUR  ROLE(S)  IN  THE  MAP. 

(CHECK  APPROPRIATE  ITEM(S). ) 

_  DEFINE  OR  REVIEW  REQUIREMENTS,  STANDARDS,  CRITERIA 

_ DEVELOP  OR  MONITOR  THE  DESIGN  OF  EMERGING  SYSTEM  CONCEPTS 

_ DESIGN  OR  MONITOR  THE  CHARACTERISTICS  OF  EARLY  PROTOTYPE 

SYSTEMS 

_ TEST  AND  EVALUATION  OF  SYSTEMS  (EARLY,  MID-TERM,  LATE) 

DURING  MAP. 

_  OTHER  (PLEASE  SPECIFY: _ _ 

_ ) 

2.  FOR  OR  TO  WHOM  DO  YOU  RESPOND  -  WHOSE  TASKS/DIRECTIVES  DO  YOU  USE 
TO  DO  YOUR  WORK?  (CHECK  APPROPRIATE  ITEMS) 

_ 0PM  (OFFICE  PROGRAM  MANAGER) 

_  TSM  (TRADOC  SYSTEM  MANAGER) 

_  DCD  (DIRECTORATE  OF  COMBAT  DEVELOPMENT) 

_  DOTD  (DIRECTORATE  OF  TRAINING  DEVELOPMENT) 

_  T&E  DIV/BD  (TEST  &  EVALUATION) 

_  OTHER  (PLEASE  SPECIFY: _ 

_ ) 
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PG3 

3.  WHAT  GUIDANCE  AND  ASSISTANCE  DO  YOU  USE  IN  FULFILLING  YOUR 
RESPONSIBILITY?  (CHECK  APPROPRIATE  SOURCES  AND  SPECIFY) 

A.  DOCUMENTS: 

_  DOD  # _ 

_  AR  # _ 

_  FM  # _ 

_  TM  # _ 

_  OTHER  DOCUMENTATION  (PLEASE  SPECIFY: _ 

_ _ ) 

B.  AGENCIES:  PLEASE  SPECIFY  (e.g.,  HEL,  ARI...) 


4.  TYPICALLY,  WHO  USES  THE  OUTPUT  OF  YOUR  EFFORTS  AND  PRODUCTS  ? 
(PLEASE  SPECIFY) 


PG.4 


5.  HOW  OFTEN  DO  YOU  CONSIDER  THE  FOLLOWING  PERFORMANCE  AREAS 
IN  FULFILLING  YOUR  JOB  RESPONSIBILITIES? 


(CHECK  IN  APPROPRIATE  BOXES) 

OFTEN 

SOMETIMES 

RARELY 

NEVER 

TOTAL  SYSTEM  PERFORMANCE 

SUBSYSTEM  PERFORMANCE 

OPERATOR  PERFORMANCE 

MAINTAINER  PERFORMANCE _ 

6.  HOW  OFTEN  DO  YOU  CONSIDER  THE  FOLLOWING  HUMAN  PERFORMANCE  AREAS 
IN  FULFILLING  YOUR  JOB  RESPONSIBILITY? 


(CHECK  IN  APPROPRIATE  BOXES) 

OFTEN 

SOMETIMES 

RARELY 

NEVER 

HUMAN  FACTORS 

ENGINEEERING 

MANPOWER 

PERSONNEL 

TRAINING  ; 

INDIVIDUAL  SOLDIERS 

UNIT 

SAFETY 

HEALTH  HAZARDS 

OTHER:  PLEASE  SPECIFY 
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OPERATOR/  MAINTAINER  WORKLOAD 

7.  DOES  THE  ISSUE  OF  OPERATOR  AND  MAINTAINER  WORKLOAD  (OWL)  LEVEL  GET 
CONSIDERED  IN  YOUR  WORK  ? 


Of=TEN 

SOMETIMES 

RARELY 

NEVER 

IF  EVER,  HOW  DO  YOU  ADDRESS  OWL?  (e.g.,  SPECIFIC  TOOLS,  EDUCATED  GUESSES) 


8.  WHAT  SPECIFIC  GUIDANCE  OR  DOCUMENTS  DO  YOU  NOW  USE  TO  ADDRESS  OWL? 
e.g.,  ARS,  LOCAL  REGs,  SOPs. 


9.  HOW  OFTEN  SHOULD  THE  ISSUE  OF  WORKLOAD  LEVELS  BE  CONSIDERED  IN  YOUR 
JOB? 


OFTEN 

SOMETIMES 

RARELY 

NEVER 
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PG.  6 


1 0.  HOW  WOULD  YOU  LIKE  TO  ADDRESS  OWL  ? 


1 1 .  WHAT  GUIDANCE  WOULD  YOU  LIKE  TO  HAVE  FOR  ADDRESSING  OWL?  (e.g.  POC, 
DOCUMENT...) 


1 2.  DO  YOU  FORESEE  AN  INCREASED  CONCERN  WITH  OWL  DUE  TO: 

(CHECK  APPROPRIATE  ITEM(S)) 

_  CHANGES  IN  TECHNOLOGY 

_  CHANGES  IN  REQUIREMENTS 

_  OTHER  (PLEASE  SPECIFY: _ ) 

NONE 
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BACKGROUND  INFORMATION 


YOUR  TITLE  : _ GRADE  /  RANK _ 

POSITION:  _ 

YRS.  IN  CURRENT  POSITION: _ 

AGENCY  /  UNIT:  _ _ 

SYSTEMS  INVOLVED  WITH  (NEAR  PAST,  CURRENT,  NEAR  FUTURE) 


WE  WOULD  LIKE  TO  CONTACT  PERSONS  FURTHER  ABOUT  OWL 
ISSUES.  IF  YOU  ARE  WILLING  TO  BE  CONTACTED  VIA  PHONE, 
PLEASE  FILL-IN  THE  INFORMATION  REQUESTED  BELOW. 

NAME: _  PHONE  #  : _ 
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Appendix  B 


DATE:  9FEB87 


PROGRAM  MANAGER'S  OPERATOR  WORKLOAD  ASSESSMENT  PAMPHLET 


OUTLINE 

USER  PROFILE:  The  intended  user  for  the  pamphlet  is  the  program  manager  who  is 
involved  in  both  delineating  the  needs,  and  developing  the  requirements  for  a  new  system. 
This  user  is  not  interested  in  the  details  of  workload  estimation  or  evaluation.  This  user 
also  is  not  interested  in  which  measures  or  which  techniques  offer  the  best  OWL 
assessment.  However,  what  IS  of  interest  to  the  TRADOC  or  AMC  system  manager  is 
high-level  guidance  on  what  are  the  Army  requirements  regarding  workload,  and  what 
high-level  provisions  should  be  built  into  the  system  acquisition  strategy  for  the  assessment 
of  OWL. 

FORMAT:  This  Pamphlet  will  be  structured  to  provide  a  concise,  easily  understood 
presentation  of  the  role  of  OWL  control  in  the  materiel  acquisition  process  (MAP).  Tables, 
charts,  flow  diagrams,  and  specific  examples  will  be  used  liberally  to  promote  quick 
apprehension  of  concepts. 

GOAL:  Provide  the  reader  with  an  overview  of  the  role  of  OWL  control  in  the  materiel 
acquisition  process,  including  the  nature  of  the  problem,  DoD  documents  and  requirements 
concerning  OWL  control,  and  available  technologies  to  assist  the  Army  program  manager 
in  effecting  OWL  control.  Provide  guidance  in  accessing  other  OWL  control  resources, 
especially  the  OWL  Prediction  and  Evaluation  Handbooks. 

LENGTH:  approximately  40-50  pages 
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CONTENTS 


I.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B.  Requirements,  specifications,  standards,  and  regulations  for  OWL 

C.  Impact  of  OWL  on  Army  Mission  Functions 

D.  Relationship  of  OWL  to  MANPRINT 

E.  TRADOC  Contributions  in  responding  to  OWL  concerns 

F.  AMC  Contributions  in  responding  to  OWL  concerns 

G.  Description  of  the  contents  of  this  pamphlet,  how  to  use  this  pamphlet 

STRATEGY:  Introduce  the  user  to  the  key  OWL  concepts  and  regulations.  Provide  the 
proper  framework  on  how  to  use  this  handbook. 


II.  OVERVIEW  OF  OWL  FUNDAMENTALS 

A.  OWL  Performance  Model 

B.  OWL  with  various  types  of  actions/behaviors 

1 .  Searching  for  and  receiving  information 

2.  Identifying  objects,  actions,  events 

3.  Problem  solving 

4.  Decision  making 

C.  OWL  considerations  for  system  types 
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1 .  Aviation 


2.  C3I 

3.  Air  defense 

4.  Armoied/Mechanized  operations 

5.  Maintenance 

6.  Supply 

D.  OWL  considerations  during  the  Materiel  Acquisition  Process  (MAP) 

E.  OWL  Assessment  Program 

1 .  Prediction  (analytic  approach) 

2.  Evaluation  (empirical  approach) 

3.  Analysis  of  results 

F.  OWL  Control  Plan 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  key  areas  and  steps 
involved  in  OWL  prediction/evaluation  and  its  relationship  to  the  materiel  acquisition 
process. 


Ill,  OWL  IN  REQUIREMENTS  ANALYSIS/CONCEPT  FORMULATION 

A.  TRADOC  perspective 

B .  AMC  perspective 

C.  OWL  trade  offs  in  concept  formulation 

D.  Development  of  a  preliminary  OWL  Control  Plan 

E.  Key  OWL  resources 

1 .  Documents 

2.  Organizations  (e.g,  HEL,  ARI) 


B-3 


3.  Individuals  (i.e.,  HFE  specialists) 

F  The  TRADOC  System  Manager’s  (TSM)  OWL  Concept  Formulation  Check  List 

1.  What  should  the  TSM  be  ensuring  is  accomplished. 

G .  The  AMC  Program  Manager's  (PM)  OWL  Concept  Formulation  Check  List 
1 .  What  should  the  PM  be  ensuring  is  accomplished. 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  requirements  analysis  and  concept  formulation.  Provide  the  user 
the  know-how  to  inte  grate  the  OWL  Control  Plan  with  existing  requirements,  programs, 
and  other  control  plans  that  are  part  of  the  materiel  acquisition  process. 


IV.  OWL  IN  SYSTEM  DEVELOPMENT 

A.  The  OWL  Control  Plan 

B.  Methods  for  assessment 

1 .  OWL  Prediction  (analytic  approach) 

2.  OWL  Evaluation  (empirical  approach) 

C.  Key  OWL  resources 

1.  Documents 

2.  Organizations  (e.g.,  HEL,  ARI) 

3.  Individuals  (i.e.,  HFE  specialists) 

D.  TSM  OWL  Check  List  for  OWL  Prediction 

E.  TSM  OWL  Check  List  for  OWL  Evaluation 

F .  AMC  PM  OWL  Check  List  for  OWL  Prediction 

G .  AMC  PM  OWL  Check  List  for  OWL  Evaluation 
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STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  Assessment  Program  with  existing  requirements,  programs,  and  other 
control  plans  that  are  part  of  the  materiel  acquisition  process. 


V.  ITERATIVE  NATURE  OF  OWL  ASSESSMENT 

A.  Materiel  acquisition  process 

B.  System  design  decisions 

C.  Evolution  of  OWL  considerations 

STRATEGY:  Establish  the  concept  that  the  OWL  Assessment  Program  and  its  management 
and  control  are  evolving  processes  which  are  modified  as  the  materiel  acquisition  process 
progresses. 


VI.  EXAMPLE 

A.  An  example  will  be  provided  that  delineates  both  TSM  and  AMC  PM  development 
and  implementation  of  their  respective  OWL  Control  Plans. 

STRATEGY:  Provide  the  user  with  a  concrete  example  of  an  OWL  Control  Plan. 


VII.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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Appendix  C 


DATE:  9FEB87 


WORKLOAD  PREDICTION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Prediction  Handbook  is  intended  for  the  system  designer 
during  the  concept  and  early  design  phases  of  the  military  system  acquisition  cycle.  This 
user  is  interested  in  the  different  OWL  measures  and  techniques  applicable  during  early 
design.  This  user  is  typically  the  person  who  (1)  makes  the  decision  of  which  OWL 
assessment  tools  to  use,  and  (2)  adapts  those  tools  to  fit  the  specific  needs  and 
characteristics  for  the  system  of  interest.  To  perform  these  functions,  the  system  designer 
will  identify  the  system  requirements  and  specific  design  objectives  for  which  the  workload 
assessment  methodology  is  needed  and  will  use  "the  matching  model"  procedure  to  select 
an  optimal  OWL  Assessment  Battery.  The  handbook  provides  guidance  on  the 
implementation  of  the  OWL  Assessment  Battery. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
user  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an  OWL 
assessment  program  during  early  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  assessment  program  with  existing  requirements,  programs,  and 
methodologies  that  are  part  of  the  materiel  acquisition  process  (MAP). 

LENGTH:  approximately  75-100  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  What  is  Operator  Workload  (OWL)? 

B.  Requirements,  specifications,  standards,  and  regulations  for  OWL 

C.  System  performance  and  operator  workload  (system  requirements) 

D.  Operator  workload  performance  model:  variables/factors  to  consider 

E.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  operator 
workload  assessment  program  during  concept  and  preliminary  design  phases 

F.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  key  OWL  concepts  and  requirements.  Orient  the  user  to 
the  critical  concept  -  the  OWL  Performance  Model  -  that  underlies  the  methodology  for 
workload  prediction.  Provide  the  framework  on  how  to  use  the  Handbook. 


II.  OVERVIEW  OF  OPERATOR  WORKLOAD  PREDICTION 

A.  What  is  OWL  prediction? 

B.  The  relationship  between  OWL  prediction  &  OWL  evaluation 

C.  Factors  to  consider  for  OWL  prediction 

1. System  requirements,  (i.e.,  system  performance) 

2.0perator  capabilities/skills/behaviors  required 
3. Stage  in  materiel  acquisition  process 
4.0WL  performance  model 


5.0WL  assessment  techniques 


D.  Matching  model  for  establishing  an  OWL  Assessment  Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
prediction  (analytic  approach)  and  its  relationship  to  OWL  evaluation  (empirical  approach) 


III.  OPERATOR  WORKLOAD  PREDICTION  METHODOLOGY 

A.  Determine  system  performance  requirements 

1.  Justification 

a.  JMSNS  (Justification  of  Major  Systems  New  Starts) 

b.  MIL-H-46855B 

2.  Requirement  source  documents  (cf.,  DoD  Directive  5000.2,  AR  15-14, 
AR  70-1,  AR  70-10,  AR  71-3) 

B .  Determine  operator  actions/behaviors  for  system  usage 

1.  Sources 

a.  Requirement  documents 

b.  Task  analyses 

c.  Expert  opinions 

d.  Comparative  systems 

e.  Mock-ups 

f.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system 
usage(cf..  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell, 
&  Shearer,  1964) 

-  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 
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c.  Problem  solving 

d.  Decision  making 


C.  Determine  stage  in  materiel  acquisition  process 

1 .  Mission  area  analysis  and  JMSNS 

2.  Concept  and  exploration  phase 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
(decision  making  process) 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  battery 

2.  Decision  tables  for  workload  methodologies 

F.  Analytic  assessment  methods 

1 .  Purpose:  Prediction  of  OWL  &  "chokepoints" 

2.  Sensitivity 

3.  Representation  of  OWL  issues/behaviors 

4.  Techniques 

a.  Expert  opinions 

b.  Comparisons 

c.  Simulations/Models 

d.  Math  models 

e.  Task  analytic  methods 

5.  Interpretation  of  assessment  results 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an 
OWL  Assessment  Program.  Provide  the  user  the  know-how  to  integrate  the  OWL 
Assessment  Program  with  existing  requirements,  programs,  and  methodologies  that  are 
part  of  the  materiel  acquisition  process. 
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IV.  ITERATIVE  NATURE  OF  OWL  PREDICTION 


A.  Materiel  acquisition  process 

B .  System  design  decisions 

C.  etc.. 

STRATEGY:  Establish  the  concept  that  the  OWL  assessment  program  (developed  from 
using  this  Handbook)  is  an  evolving  program  which  will  be  modified  as  the  materiel 
acquisition  process  progresses. 


V.  EXAMPLES 

A.  Examples  will  be  provided  for  each  of  the  assessment  techniques. 

STRATEGY:  Provide  the  user  with  concrete  examples  of  OWL  assessment  programs. 
Provide  reality  to  the  OWL  prediction  methodology  described  in  the  Handbook. 


VI.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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Appendix  D 


DATE:  9FEB87 


WORKLOAD  EVALUATION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Evaluation  Handbook  is  intended  primarily  for  the 
TRADOC  community  as  well  as  system  designer  involved  with  early  developmental  testing 
(DT&E).  These  users  (e.g.,  TRADOC)  are  interested  in  interpreting  the  results  of  the 
various  workload  assessments  conducted  throughout  the  development  cycle  but  are 
primarily  concerned  with  system  evaluation.  These  evaluators  are  responsible  for  test  and 
evaluation  (T&E)  but  are  also  (like  the  designer)  interested  in  how  to  perform  OWL 
analysis.  In  addition,  they  are  more  concerned  than  the  designer  in  the  actual  constraints 
placed  by  real-world  implementation  of  a  workload  assessment.  Such  constraints  involve 
traditional  (and  non-traditional)  limits  on  testbed  resources  (e.g.,  subjects,  time,  and 
funding).  The  T&E  users  are  also  concerned  with  how  to  transform  OWL  data  and 
information  into  recommendations  for  system  design. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
reader  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an  OWL 
assessment  program.  Provide  the  user  the  know-how  to  integrate  the  OWL  assessment 
program  with  existing  requirements,  programs,  and  methodologies  that  are  part  of  the 
military  system  acquisition  cycle. 
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LENGTH:  approximately  125-150  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  What  is  Operator  Workload  (OWL)? 

B .  Requirements,  specifications,  standards,  and  regulations  for  OWL 

C.  System  performance  and  operator  workload  (system  requirements) 

D.  Operator  workload  performance  model:  variables/factors  to  consider 

E.  Purpose  for  handbook;  methodology  for  determining  and  implementing  an  OWL 
assessment  program  during  the  design  andevaluation  phases  of  the  materiel 
acquisition  process. 

F.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  the  key  OWL  concepts  and  requirements.  Orient  the 
user  to  the  critical  concept  -  OWL  Performance  Model  -  that  underlies  OWL  evaluation. 
Provide  the  proper  framework  on  how  to  use  the  handbook. 
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II.  OVERVIEW  OF  OPERATOR  WORKLOAD  EVALUATION 


A.  What  is  OWL  evaluation? 

B.  The  relationship  between  OWL  evaluation  &  OWL  prediction 

C.  Factors  to  consider  for  OWL  evaluation 

1 .  System  requirements ,  i.e.,  system  performance 

2.  OWL  assessment  program  constraints 

a.  Subjects 

b.  Time 

c.  Funding 

d.  Etc. 

3 .  Operator  capabilities/skills/behaviors  required 

4.  Materiel  acquisition  process 

5 .  Earlier  OWL  assessments  (OWL  prediction) 

6 .  OWL  performance  model 

7.  OWL  assessment  techniques 

D.  Matching  model  for  establishing  an  OWL  Assessment  Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
evaluation  (empirical  approach)  and  its  relationship  to  OWL  prediction  (analytic  approach), 
materiel  acquisition  process,  and  system  design. 

III.  OWL  ASSESSMENT  (TEST  AND  EVALUATION) 

A.  Determine  system  performance  requirements 

1.  Requirement  source  documents(cf.,  DoD  Directive  5000.2,  AR  15-14, 

AR  70-1,  AR  70-10,  AR  71-3 


B .  Determine  operator  actions/behaviors  for  system  usage 


1.  Sources 


a.  Requirement  documents 

b.  Task  analyses 

c.  Expert  opinions 

d.  Comparative  systems 

e.  Mock-ups 

f.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system  usage 
(cf.,  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell,  & 
Shearer,  1964)  -  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 

c.  Problem  solving 

d.  Decision  making 

C.  Determine  stage  in  materiel  acquisition  process 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
(decision  making  process) 

1.  For  example,  environmental  factors  such  as  noise,  vibration,  heat,  and 
cold 

2.  Sustaining  period 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  battery 

2.  Decision  tables  for  workload  methodologies 

F.  Establish  framework  for  the  evaluation 
1.  Task  scenarios 

G.  Empirical  Assessment  Methods 

1.  Purpose:  Evaluate  design  decisions 
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2.  Sensitivity 

3.  Task  scenarios:  representation 

4.  Techniques 

a.  Operator  opinion 

b.  Primary  task 

c.  Secondary  task 

d.  Physiological  responses 

5 .  Interpretation  of  assessment  results 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  implementing  an  OWL 
assessment  program  -  test  and  evaluation.  Provide  the  user  the  know-how  to  integrate  the 
OWL  assessment  program  with  existing  requirements,  programs,  and  methodologies  that 
are  part  of  the  materiel  acquisition  process. 


IV.  ITERATIVE  NATURE  OF  OWL  EVALUATION 

A.  Materiel  acquisition  process 

B.  System  design  decisions 

STRATEGY:  Establish  the  concept  that  OWL  assessment  (test  and  evaluation)  is  an 
iterative  process  that  is  conducted  throughout  the  materiel  acquisition  process  to  ensure 
OWL  is  not  a  system  problem. 


V.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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APPENDIX  E 


OWL  INFORMATION  SYSTEM 

An  OWL  Information  System  is  under  development  to  provide  for  the  control  and 
analysis  of  the  mass  of  documents  and  other  resources  comprising  the  OWL  scientific 
database.  Currently  under  implementation,  the  system  is  comprised  of  several  components. 
One  of  these  is  a  computerized  database,  to  collect,  to  organize,  to  analyze,  and  to  retrieve 
relevant  citations  and  information  associated  with  those  citations.  For  convenience,  we  will 
discuss  this  aspect  in  terms  of  the  OWL  Information  Data  Base  and  the  OWL  Analysis 
System.  The  second  part  consists  of  a  library  consisting  of  the  physical  documents  keyed 
to  the  database  for  easy  retrieval. 

Overall,  this  effort  constitutes  part  of  our  effort  for  the  development  a  set  of  useful 
tools  for  the  analysis  of  operator  workload,  providing  decision  aids,  and  an  information 
support  system  for  practitioners. 


OWL  INFORMATION  DATA  BASE 


The  data  base  contains  standard  bibliographic  information  on  each  report  or  article. 
The  software  chosen  is  dBASE  IE  which  provides  a  convenient  and  widely  used  relational 
database  and  program  system  for  IBM-PC's  and  compatible  machines.  The  hardware 
environment  was  selected  to  be  broadly  compatible  with  government  standard 
microcomputer  systems.  The  overall  system  is  composed  of  several  data  files  and  a  number 
of  accompanying  dBASE  III  programs  to  access  and  manipulate  the  data  files. 
Development  of  the  system  to  this  point  has  been  done  carefully  to  provide  flexibility  in 
retrieving  citations  based  on  particular  user  needs.  Some  additional  programs  will  be 
useful,  but  these  are  being  done  as  the  need  arises. 


The  OWL  reference  database  now  numbers  about  1400  reference  items  and  is 
growing  steadily.  There  are  several  additional  bibliographies  which  have  not  yet  been 
entered,  for  example,  one  created  by  Douglas  Aircraft  Company  which  consists  of 
approximately  700  items  and  is  supposed  to  be  available  soon  in  dBASE  IE  format.  As  we 
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review  the  actual  documents  in  the  database,  additional  references  of  interest  are  being 
collected  from  the  reviewed  documents  as  well  as  from  direct  contact  with  authors  of 
documents  already  entered. 

The  database  system  was  designed  to  access  the  information  in  a  number  of 
different  ways  to  accommodate  various  users,  those  working  on  the  project  and  more 
importantly,  those  who  will  be  in  need  of  the  information  when  the  project  is  completed. 
Each  bibliographic  record  is  composed  of  a  number  of  fields  as  shown  in  Table  E-1.  A 
form  has  been  prepared  to  facilitate  preparation  of  material  for  entry  into  the  database 
(Figure  E-1).  Of  note,  are  the  keyterm  (in  tide)  fields  which  facilitate  access  by  title. 

A  large  number  of  reports  are  possible  to  generate  citation  listings.  The  report 
programs  which  have  been  developed  to  facilitate  access  of  the  database  are: 

•  Complete  report,  alphabetized  by  authors.  This  lists  all  citations. 

•  Report  for  all  citations  for  a  single  author 

•  Report  selecting  by  keywords,  such  as  "physiological  measures" 
whereby  a  report  is  generated  that  lists  all  citations  under  this  topic. 

•  Report  listing  of  references  with  copies  in  house 

•  Several  utility  reports  for  editing,  etc. 

These  report  programs  currently  provide  bibliographies  in  two  standard  forms:  Human 
Factors  Society  and  American  Psychological  Association.  The  programs  are  developed  so 
that  others  could  be  added  easily.  Additionally,  several  programs  have  been  developed  to 
enter  and  edit  the  reference  file  along  with  necessary  utility  programs.  The  development  of 
these  programs  allows  the  user  to  enter  additional  references  and  reviews  and  still  maintain 
the  system  easily. 


OWL  DEFORMATION  ANALYSIS  SYSTEM 

While  the  bibliographic  file  will  be  of  general  use,  the  more  important  part  of  the 
data  base  consists  of  a  separate  file  containing  the  reviews  produced  by  members  of  the 
Analytics,  Inc.  OWL  team.  The  reviews  are  based  on  objective  and  standard  classification 
techniques.  The  review  form  is  shown  in  Figure  E-2  and  the  structure  of  the  review  data 
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file  is  shown  in  Table  E-2.  This  analysis  follows  the  taxonomy  presented  by  Wierwille  and 
Williges  (1978)  which  was  in  turn,  modeled  on  the  taxonomy  developed  earlier  by 
Berliner,  Angell,  and  Shearer  (1964).  We  have  augmented  this  analysis  with  the  addition 
of  statistical  and  observer/subject  categories,  some  of  which  were  suggested  from  an 
analysis  done  by  Douglas  Aircraft  Company.  Ultimately,  most  papers  will  be  reviewed. 

The  reviews  are  entered  into  a  second,  separate  data  file,  linked  to  the  reference  file 
by  item  number  (and  first  author  as  a  check).  (Item  No.  is  an  arbitrary  number  assigned 
principally  on  order  of  entry.)  The  use  of  separate  files  permits  multiple  reviews  (several 
reviewers)  on  a  given  paper  and  takes  advantage  of  the  relational  database  properties  of 
dBASE  in.  To  date,  reviews  for  approximately  100  papers  have  been  entered;  this  aspect 
will  receive  considerable  attention  in  the  near  future  as  we  enter  into  Task  3. 

As  the  number  of  reviews  entered  into  the  system  increases,  we  will  develop  a 
report  generator  to  permit  the  user  easy  access  using  the  information  contained  in  the 
reviews. 


OWL  LIBRARY 

The  OWL  Library  is  being  assembled  through  both  formal  and  informal  sources. 
The  sources  include  publications  available  through  DTIC  and  NTIS,  journal  articles, 
conference  proceedings,  conference  papers,  monographs,  dissertations,  etc. 

Additionally,  documents  are  being  sought  directly  from  well  established  workload 
laboratories,  e.g.,  NASA  Ames  and  NASA  Langley,  Air  Force  AMHRL,  etc.  Also,  letters 
are  being  sent  directly  to  authors  already  in  the  database  to  solicit  copies  of  their  work  as 
well  as  any  work  which  is  new  or  was  inadvertently  missed  on  the  first  pass  through  other 
sources.  Currently  we  are  receiving  20  to  30  documents  a  week  from  authors  with 
approximately  20%  being  additions  to  the  reference  database. 
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TABLE  E-1 


Listing  and  description  of  fields  in  OWL  reference  database 


Description _ ElELD  NAME _ TYPE  LENGTH _ DECIMAL 


Owl  database  # 

ITEMNO 

N 

8 

Review  done 

REVIEWED 

C 

1 

Lib.  Catelog  # 

CATALOG 

C 

12 

Report  No. 

REPORTNO 

c 

25 

Title  -  Article 

ARTICLE 

c 

250 

Title  -  Book 

BOOKTITLE 

c 

250 

First  Author 

AUTHORl 

c 

25 

Second  Author 

AUTHOR2 

c 

25 

Third  Author 

AUTHORS 

c 

25 

Fourth  Author 

AUTHOR4 

c 

25 

Fifth  Author 

AUTHOR^ 

c 

25 

Corp.  Author 

CORPAUTHOR 

c 

100 

Vol.  Editor 

EDITOR 

c 

60 

Book  Co. 

PUBLISHER 

c 

80 

Journal 

JOURNAL 

c 

80 

Year  of  Pub. 

PUB YEAR 

c 

4 

Volume 

VOLUME 

c 

8 

Pages 

PAGES 

c 

13 

Keyterm 

KEYTERMl 

c 

50 

Keyterm 

KEYTERM2 

c 

50 

Keyterm 

KEYTERM3 

c 

50 

Keyterm 

KEYTERM4 

c 

50 

Keyterm 

KEYTERM5 

c 

50 

Do  we  have? 

INHOUSE 

c 

1 

Notes 

NOTES 

c 

250 

Date  entered 

DATE 

c 

8 
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HGUREE-l 


OWL  REFERENCE  ENTRY  FORM 


Item  No. _ (Assigned  by  computer) 


Author  1 :  _  Author!  : 

Author  3  :  _  Author  4  : 

Author  5  :  _ 

CORP.  AUTHOR: _ 

(Use  if  no  authors) _ 

TITLE:(Joumal,  Article,  Book,  or  Chapter) 


EDITOR(s): _ 

BOOKTITLE(For  editted  books,proceed.): 
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REPORT  NO. _ 

JOURNAL:^ _ _ _ 

PUBLISHER:  (Use  only  if  Booktitle  used;  Place  &  pub.  company) 

Year  of  Publication: _  Volume  (&  No.): _ ( _ )  Pages: 

Keyterm  1  _ 

Keyterm  2  : _ _ _ _ — 

Keyterm  3  : _ _ _ _ 

Keyterm  4 _ _ _ 

Keyterm  5  : _ _ _ 

Copy  of  Paper  in  house?  _  (Y  or  N) 

Notes:(250  characters) 
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Table  E-2. 


Description 

FIELDNAME 

TYPE 

LEN 

DECIMAL 

GENERAL 

OWL  Database  # 

ITEMNO 

N 

8 

3 

First  Author 

AUTHORl 

C 

25 

Reviewer 

REVIEWER 

C 

20 

% 

Review  rating 

RATING 

c 

2 

REPORT  NATURE  or  TYPE  (TYPE) 

DoD 

TYPEl 

c 

1 

Theoretical 

TYPE2 

c 

1 

Review 

TYPES 

c 

1 

Bibliographic 

TYPE4 

c 

1 

Methodological 

TYPES 

c 

1 

Lab  Experimentation 

TYPE6 

c 

1 

System  application 

TYPE? 

c 

1 

Specific  system 

TYPE7S 

c 

25 

FIDELITY  (HDEL) 

Actual  system 

FIDELl 

c 

4 

Simulation  FIDEL2 

Applied  Lab  FIDEL3 
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Basic  Lab 


FIDEL4 


ANALYTIC  TAXONOMY  (TAXA) 


Expert  Opinion 

TAXAl 

C 

1 

Comparison 

TAXA2 

C 

1 

Simulation  Models 

TAXA3 

c 

1 

Math  Models 

TAXA4 

c 

1 

Manual 

TAXA41 

c 

3 

Info  Theor. 

TAXA42 

Other 

TAXA43 

Task  Analytic 

TAXA5 

c 

1 

EMPIRICAL  TAXANOMY  (TAKE) 


Primary  task 

TAXEl 

C 

1 

Performance  rel 

TAXEll 

C 

3 

Strategy  rel. 

TAXE12 

Other 

TAXE13 

Secondary  task 

TAXE2 

c 

1 

Subsid.  task 

TAXE21 

c 

4 

Probe  task 

TAXE22 

Dual  taks 

TAXE23 

Other 

TAXE24 
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Subjective  scales 

TAXES 

C 

1 

Rating  Scales 

TAXE31 

C 

5 

CH 

TAXES  11 

c 

5 

MCH 

TAXES  12 

SWAT 

TAXES  IS 

NASA  Biploar 

TAXES  14 

Other 

TAXES  15 

Questionaire 

TAXES2 

Interviews 

TAXESS 

Other 

TAXES4 

Physiological 

TAXE4 

c 

1 

Heart  rate 

TAXE41 

c 

10 

Heart  rate 

TAXE411 

c 

1 

(0.1  Hz.) 

Eye  movements 

TAXE42 

Respiration 

TAXE4S 

Blood  Pressure 

TAXE44 

GSR  (Skin) 

TAXE45 

EMG  (Muslce) 

TAXE46 

EEG  (Brain  Act.) 

TAXE47 

EP  (Evoked  Pot.) 

TAXE48 

Body  Fluid 

TAXE49 

Other 

TAXE40 

WORKLOAD  CHARACTERISTICS  (WC) 


1 
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Type  -  Mental  (TM) 

WCTMl 

C 

1 

Degree 

WCTMll 

c 

2 

Duration 

WCTM12 

Type  -  Psysical  (TP)WCTPl 

c 

1 

Degree 

WCTPll 

c 

2 

Duration 

WCTP12 

Function  (FUN)  ** 

Navigation 

WCFUNl 

c 

10 

Communications 

WCFUN2 

Command  decision 

WCFUN3 

Open  &  Monitor 

WCFUN4 

Collision  avoid 

WCFUN5 

Path  control 

WCFUN6 

WCFUN7 

WCFUN8 

WCFUN9 

WCFUNO 

Factors  (FAC) 

Normal  Oper 

WCFACl 

c 

3 

Time  Pressure 

WCFAC2 

Abnormal 

WCFAC3 

Specify 

WCFAC3S 

c 

25 

TYPE  of  SUBJECT  (TYPES) _ 

Expert  TYPES  1  C  4 
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Novice 

Student 

Other 


TYPES2 

TYPES3 

TYPES4 


TASK  DESCRIPTION  (TASK) 

Task  1  description 

TASKl 

C 

23 

Discrete 

TASKll 

C 

3 

Paced 

TASK12 

Continuous 

TASK13 

-  Response  Form  (RF) 

Verbal 

TASKRFll 

c 

4 

Discrete  motor 

TASKRF12 

Cont.  motor 

TASKRF13 

Other 

TASKRF14 

-  Response  Measure  (RM) 

Time 

TASKRMll 

c 

4 

Accuracy/error 

TASKRM12 

Event 

TASKRM13 

Other 

TASKRM14 

Task  2  description 

TASK2 

C 

23 

Discrete 

TASK21 

C 

3 

Paced  TASK22 
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Continuous 

TASK23 

-  Response  Form  (RF) 

Verbal 

TASKRF21 

C 

4 

Discrete  motor 

TASKRF22 

Cont.  motor 

TASKRF23 

Other 

TASKRF24 

-  Response  Measure  (RM) 

Time 

TASKRM21 

C 

4 

Accuracy/error 

TASKRM22 

Event 

TASKRM23 

Other 

TASKRM24 

Task  3  description  TASK3 

C 

23 

Discrete 

TASK31 

C 

3 

Paced 

TASK32 

Continuous 

TASK33 

-  Response  form  (RF) 

Verbal 

TASKRF31 

c 

4 

Discrete  motor 

TASKRF32 

Cont.  motor 

TASKRF33 

Other 

TASKRF34 

-  Response  Measure  (RM) 

Time 

TASKRM31 

c 

4 

Accuracy/error 

TASKRM32 

Event 

TASKRM33 

Other 

TASKRM34 
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Task  4  description 

TASK4 

C 

23 

Discrete 

TASK41 

C 

3 

Paced 

TASK42 

Continuous 

TASK43 

-  Response  Form  (RF) 

Verbal 

TASKRF41 

c 

4 

Discrete  motor 

TASKRF42 

Cont.  motor 

TASKRF43 

Other 

TASKRF44 

-  Response  Measure  (RM) 

Time 

TASKRM41 

c 

4 

Accuracy/error 

TASKRM42 

Event 

TASKRM43 

Other 

TASKRM44 

METRIC  QUALITIES  (METRIC) 


Indiv.  Differences 

METRICl 

C 

1 

Reliability 

METRICll 

C 

3 

Stability 

METRIC12 

Other  S  diff 

METRIC13 

Comparitive  Sensit. 

METRIC2 

C 

1 

Validity 

METRIC3 

C 

1 
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-  Task  to  task 

Sub  to  sub 

METRICS  1 

C 

3 

Sub  to  obj 

METRIC32 

Obj  to  obj 

METRIC33 

-Type 

Content 

METRIC34 

C 

3 

Construct 

METRIC35 

Predictive 

METRIC36 

Reliability 

METRIC4 

C 

1 

Test  -  retest 

METRIC41 

C 

4 

Split  half 

METRIC42 

Alternate  forms 

METRIC43 

Interrater 

METRIC44 

KEYWORDS  (KEYTERM) 


KEYl 

KEYTERMl 

C 

30 

KEY2 

KEYTERM2 

C 

30 

KEYS 

KEYTERM3 

C 

30 

KEY4 

KEYTERM4 

C 

30 

COMMENTS  (COMMENT) 

Comment  COMMENTl 

C 

75 
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Comment 

COMMENT2 

C 

75 

Comment 

COMMENT3 

C 

75 

Comment 
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Due  to  the  limitation  of  a  maximum  of  128  fields  in  dbase  III,  a  number  of  the 
characteristics  had  to  be  combined  into  a  single  field.  The  rules  for  this  are  based  on  the 
estimated  probability  of  searching  within  a  category.  For  example,  Math  models  from  the 
analytic  taxonomy  is  a  single  field.  The  subcategories  under  math  models  are  a  combined 
field.  If  Math  Models  is  entered  (something  other  than  blank)  then  you  would  be  asked  for 
the  subcategories  and  these  would  be  stored  in  TAXA41,  in  order.  Thus,  if  Infomation 
Theoretic  were  the  only  subcategory,  the  contents  of  TAXA41  would  look  like  this  '.X.'. 
(Periods  are  put  into  the  field  when  there  is  no  entry.  This  is  done  so  that  if  the  whole  field 
is  printed  it  is  easy  to  tell  where  the  entry  is.) 


In  the  main  table,  combined  entries  are  indented.  Those  fields  which  are  combined 
into  another  label  are  blank  for  field  type  and  length.  The  example  below  illustrates  this. 
TAXA42  and  TAXA43  are  combined  into  TAXA41. 


Math  Models  TAXA4 


C  1 


Manual  TAXA41  C  3 

Info  Theor.  TAXA42 
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Other 


TAXA43 


Further,  as  the  indentation  above  represents,  an  indented  field  will  be  empty 
automatically  if  the  main  field  is  blank.  Thus,  if  TAXA4  is  blank,  the  subcategories  will 
also  be  empty  e.g.,  TAXA41  would  be 


In  a  case  where  there  are  subsubcategories,  these  are  listed  in  the  the  logical  order 
following  the  subcategory  driving  it.  TAXE31  has  a  subcategory  of  TAXE311,  ...,. 
TAXE311  is  a  field  containing  these  entries  and  follows  TAXE31  on  the  dbase  III  file 
listing.  TAXE32,  etc.  is  stored  in  field  TAXE31. 
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EXECUTIVE  SUMMARY 


This  report  is  one  of  a  series  describing  a  program  for  the  development  and 
validation  bf  a  methodology  for  estimation  and  evaluation  of  operator  workload  (OWL)  in 
Army  systems.  It  presents  the  results  of  Task  2  of  Analytics'  contract  with  the  Army 
Research  Institute  (ARI)  to  "Identify  Army  Requirements  Regarding  OWL,  Select  Specific 
Army  Systems  to  Analyze  ,  and  Provide  Outline  of  OWL  Products".  Included  are  the 
results  of  component  subtasks;  (2.1)  Review  Army  Requirements  and  Reports,  (2.2) 
Assess  User  Needs,  (2.3)  Outline  Final  Products,  and  (2.4)  Select  Prototype  Army 
Systems  to  Evaluate.  The  overall  purpose  of  this  report  is  to  characterize  existing  and 
future  Army  requirements  and  needs  for  OWL  assessment,  to  tailor  the  OWTL  program  to 
meet  these  requirements  and  needs,  and  to  identify  emerging  Army  systems  that  are 
appropriate  candidates  for  exercising  families  of  OWL  assessment  techniques. 

Based  on  the  review  of  Army  documents  and  regulations,  there  seemed  to  be  a  void 
in  specific  guidance  concerning  the  implementation  of  OWL  assessment  during  the  Materiel 
Acquisition  Process  (MAP).  Such  lack  of  specific  guidance  concerning  OWT.  assessment 
was  further  substantiated  in  our  interviews  as  well  as  from  questionnaire  data  with  Army 
personnel  who  play  integral  roles  during  the  MAP.  Our  assessment  of  these  findings  has 
resulted  in  tailoring  the  proposed  products  (e.g,.  Outlines  of  OWL  Handbooks  and 
Pamphlet)  to  meet  the  apparent  need  for  OWL  guidance  throughout  the  MAP.  In  addition, 
recommendations  are  offered  in  the  report  for  integrating  our  efforts  with  existing  Army 
programs  (e.g.,  MANPRINT)  to  assure  that  OWL  receives  adequate  consideration 
throughout  the  MAP.  With  respect  to  selecting  prototype  Army  systems  to  evaluate, 
candidate  systems  are  offered  that  allow  a  wide-range  of  OWL  techniques  to  be  employed 
as  well  as  providing  opportunities  to  make  substantial  and  positive  contributions  toward 
impacting  the  design  of  these  systems . 
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1.  INTRODUCTION 


This  report  is  one  of  a  series  describing  a  program  for  the  development  and 
validation  of  a  methodology  for  estimation  and  evaluation  of  operator  workload  (OWL)  in 
Army  systems.  It  presents  the  results  of  Task  2  of  Analytics'  contract  with  the  Army 
Research  Institute  (ART).  The  components  comprising  this  task  are: 

•  Review  Army  Requirements 

•  Assess  User  Needs 

•  Outline  Final  Products 

•  Select  Prototype  Army  Systems  to  Evaluate 

The  overall  purpose  of  this  report  is  to:  Present  Army  requirements  and  needs  as 
obtained  throughout  document  review  and  interviews  regarding  OWL  issues  and  concerns, 
outlines  of  final  products  based  on  those  requirements  and  needs,  and  to  suggest  emerging 
Army  systems  for  OWL  evaluation. 


1  ■  1  Overview  of  Program  Progress 

The  OWL  program  is  one  of  several  focusing  on  the  practical  problem  of 
determining  what  the  Army  can  and  should  do  to  assure  that  systems  can  be  adequately 
operated  by  prospective  personnel.  As  new  technologies  are  implemented  with  greater 
degrees  of  computer-interaction,  there  is  growing  concern  questioning  the  ability  of 
soldiers  to  operate  these  systems.  With  regard  to  OWL,  pertinent  questions  include,  "How 
much  information  processing,  decision  making  and  other  cognitive  tasks  can  system 
operators  handle?"  and  "Under  what  time  and  other  limitations  may  operators  continue  to 
function  before  overload  and  performance  degradations  occur?"  These  OWL  questions 
need  to  be  addressed  during  system  development  to  avoid  marginal  or  inoperable  systems. 

The  primary  focus  of  the  present  program  is  with  single  operator  workload.  For 
present  purposes,  OWL  may  be  thought  of  as  a  representation  via  predictive  and  empirical 
assessment  techniques  of  a  human  operator's  relative  limitation  in  the  capability  to  perform 
work.  Here,  relative  limitation  implies  a  functional  relation  between  (1)  actual  operator 
performance  in  the  context  of  mission  requirements  and  (2)  the  operator's  ultimate 
performance  capability.  Our  emphasis  is  on  the  cognitive,  perceptual  and  psychomotor 
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aspects  of  workload,  although  it  is  recognized  that  there  are  physical  workload  issues  more 
associated  with  maintainer  tasks  (lifting,  carrying,  etc.). 

A  necessary  step  in  the  process  of  providing  practical  assistance  to  the  Army  is  to 
understand  current  procedures  and  requirements  for  addressing  operator  workload  in  the 
Materiel  Acquisition  Process  (MAP).  This  understanding  will  form  the  basis  for 
development  of  useful  guidance  for  the  Army  community.  To  gain  this  understanding  two 
avenues  were  employed.  First,  the  procedural  aspects  of  the  Army  MAP  were  reviewed  via 
written  documents.  Second,  discussions  were  conducted  with  military  and  civilian 
personnel  within  the  Department  of  the  Army  (DA)  to  further  understanding  of  their  roles 
in  the  MAP,  their  current  methods  of  addressing  OWL,  and  their  thoughts  on  what  could 
be  provided  to  them  that  would  be  most  useful  in  their  jobs.  This  process  of  understanding 
and  its  results  are  discussed  later  in  the  report. 

From  the  comments  and  suggestions  gathered,  draft  outlines  of  the  three  handbooks 
have  been  developed  for  this  report.  The  handbooks  which  will  be  developed  during  a  later 
effort  (Task  5)  will  form  the  basis  of  the  guidance  provided  to  the  Army  users  for 
addressing  operator  workload  The  guidance  will  result  from  a  critical  review  of  the  ciurent 
scientific  literature  concerning  definition,  prediction,  measurement  and  evaluation  of 
workload  which  will  be  accomplished  as  part  of  the  follow-on  in  Task  3.  The  suggested 
guidance  and  methodologies  will  be  validated  subsequently  by  application  to  specific  Army 
systems  (Task  4).  However,  the  systems  have  been  chosen  as  part  of  the  present  effort  so 
that  familiarization  with  system  specifics  and  necessary  coordination  with  the  appropriate 
Army  organizations  can  begin.  As  will  be  seen,  the  task  of  choosing  prospective  systems 
was  assisted  by  the  interview  process;  those  most  familiar  with  what  systems  are  most 
appropriate  were  some  of  the  same  individuals  with  whom  we  were  speaking  about  OWL 
requirements  and  needs.  The  application  of  the  techniques  to  the  three  selected  systems 
will  serve  as  the  means  to  validate  the  practical  approach  being  devised  as  well  as  providing 
benefits  for  the  systems  and  the  Army. 


1.2  Oreanization  of  Report 


The  body  of  this  report  presents  the  results  of  Task  2.  Section  2,  in  particular, 
discusses  the  review  of  documents  that  describe  the  Army  Materiel  Acquisition  Process 
and  how  operator  workload  is  currently  addressed  within  the  MAP.  Subsequently,  the 
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results  of  the  interviews  of  members  of  the  Army  community  concerning  their  roles  and 
responsibilities  (both  current  and  projected)  in  the  MAP  and  their  viewpoints  concerning 
OWL  are  presented  in  Section  3.  In  Section  4,  the  concepts  and  rationale  for  the  final 
products  are  discussed  and  the  product  outlines  are  presented.  The  descriptions  of  Army 
systems  that  have  been  selected  for  assessment  and  validation  of  OWL  assessment 
techniques  are  presented  in  Section  5.  Finally,  Sections  6  and  7  discuss  other  progress  to 
date  as  well  as  plans  for  future  tasks  of  the  OWL  program. 


2.  REVIEW  OF  ARMY  REQUIREMENTS 


2.1  Introduction 


The  Army  maintains  a  continuous  effort  to  field  the  most  capable  force  possible. 
When  changes  in  doctrine,  organization  or  training  cannot  eliminate  identified  deficiencies, 
then  a  decision  may  be  made  to  eliminate  the  deficiencies  through  equipment  acquisition 
(i.e.,  procuring  hardware  systems).  The  process  by  which  new  equipment  is  conceived, 
designed,  developed  and  procured  is  given  in  regulations,  directives,  and  other 
documents. 

As  part  of  the  OWL  project  effort,  a  review  of  the  documents  and  regulations  that 
drive  Army  materiel  acquisition  was  performed.  The  review  was  one  method  to  gain  an 
understanding  of  the  Army  system  of  materiel  acquisition  —  the  way  the  requirements  for 
materiel  are  developed,  the  information  needed  to  develop  requirements,  how  the 
requirements  are  translated  to  hardware  design  and  development,  and  the  roles  of  members 
of  the  Army  community  who  manage  or  perform  these  tasks.  The  document  review  also 
was  a  means  of  identifying  guidance  inconsistencies  and  voids. 

This  chapter  describes  the  approach  used  to  understand  the  Army  acquisition 
process,  how  the  issue  of  OWL  is  currently  addressed  in  the  process,  what  documents 
provide  guidance  and  where  we  see  OWL  issues  potentially  being  addressed  in  the 
acquisition  process. 


2.2  Approach 

A  series  of  documents  were  reviewed  for  two  major  purposes.  First,  we  needed  to 
assure  a  comprehensive  understanding  of  the  documented  Army  Materiel  Acquisition 
Process  (MAP).  For  this  purpose,  a  detailed  review  of  both  directly  and  indirectly  relevant 
documents  was  conducted  to  identify  (1)  if  and  where  OWL  issues  are  considered  and, 
(2)  how  they  are  currently  addressed. 

The  process  was  an  iterative  one  —  as  more  relevant  documents  were  identified, 
ones  read  previously  were  reread  for  increased  information  and  understanding. 
Additionally,  the  interview  and  questionnaire  methodology  employed  to  assess  Army  user 


needs  (see  Section  3.)  yielded  information  that  added  to  our  understanding  of  both  the 
MAP  and  associated  issues  regarding  the  assessment  of  OWL.  The  interviews  provided 
for  iterative  document  review  as  well  as  further  contacts. 

The  following  sections,  then,  will  present  discussion  of  our  review.  The 
discussion  of  the  Army  Materiel  Acquisition  Process  in  Section  2.3  deals  with  the  process 
as  described  in  Army  Regulations  (ARs)  and  other  written  documents.  Of  particular 
importance  is  the  recent  emphasis  on  the  Army  Streamlined  Acquisition  Process  (ASAP) 
and  other  acquisition  alternatives,  such  as  Non-developmental  Items  (NDI)  and  product 
improvements  (P3I,  PIP).  Section  2.4  describes  the  few  places  in  the  reviewed  documents 
where  OWL  is  mentioned.  In  addition,  the  review  was  expanded  to  include  mention  of 
Human  Factors  Engineering  (HFE)  and  MANPRINT  where  OWL  concerns  may  be 
expected  to  be  of  greatest  interest  and  importance.  Finally,  several  conclusions  are  drawn 
from  the  document  review  of  OWL  and  recommendations  are  made. 

Some  documents  are  more  important  than  others  and  specific  reference  will  identify 
those  when  appropriate.  In  all  cases,  the  documents  reviewed  were  the  latest  editions  that 
could  be  obtained.  It  is  noteworthy,  as  can  be  seen  by  the  effective  dates  of  many  of  the 
ARs,  much  guidance  has  been  recently  revised  and  published.  In  one  sense  then,  this 
discussion  is  based  on  guidance  available  within  the  present  slice  of  time.  However,  the 
general  process  of  identifying  requirements  and  developing/acquiring  equipment  to  fulfill 
the  requirements  remains  essentially  unchanged.  The  discussions  that  follow  will  make 
specific  reference  to  the  latest  guidance  available,  but  will  also  include  an  overall 
perspective  of  the  Army  MAP  and  OWL  issues  in  MAP. 

2.3  The  Army  Materiel  Acquisition  Process 


2.3.1  Introduction 


There  has  been  a  great  deal  of  innovation  and  adjustment  in  the  acquisition  process 
during  this  decade.  Incorporation  of  the  recommendations  of  bodies  such  as  the  Packard 
Commission  have  resulted  in  many  changes  in  the  way  the  Department  of  Defense  and  the 
Army  approach  developing  and  fielding  new  materiel.  Four  broad  categories  of  acquisition 
methods  are  recognized.  These  are; 
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•  Traditional  Process 

•  Army  Streamlined  Acquisition  Process  (ASAP) 

•  Non-Development  Items  (NDI) 

•  Product  Improvement. 

The  traditional  process  is  described  in  a  degree  of  detail  which  may  seem  out  of 
proportion  to  its  modem  application.  This  approach  is  taken  because  most  persons  with 
some  experience  in  materiel  acquisition  are  familiar  with  the  traditional  process  and  use  it 
for  comparisons.  The  other  development  methods  are  described  using  the  traditional 
process  as  a  baseline.  This  section  describes  the  Army  Materiel  Acquisition  Process 
(MAP)  and  projects  how  OWL  considerations  could  or  should  be  integrated  into  the  MAP. 

Major  reference  sources  for  this  section  are  AR  70-1,  Systems  Acquisition  Policy 
and  Procedures.  AR  71-9,  Materiel  Objectives  and  Requirements  and  DARCOM  (now 
AMC)  /  TRADOC  Pamphlet  70-2,  Materiel  Acquisition  Handbook.  Major  implementing 
regulations  in  specific  disciplines,  such  as  AR  70-10,  Test  and  Evaluation,  and  AR  602-2 
MANPRINT  have  also  been  useful  sources.  AMC  Regulation  70-52,  System 
Engineering,  is  a  relatively  new  publication  which  may  have  impact  on  incorporating  OWL 
enhancements  to  new  or  improved  materiel.  It  provides  renewed  emphasis  to  the 
importance  of  the  systems  engineering  process  in  materiel  development.  Department  of  the 
Army  Pamphlet  11-25,  Life  Cycle  System  Management  Model  for  Army  Systems,  has 
also  been  useful.  This  latter  reference  is,  howeyer,  over  ten  years  old  and  must  be  used 
with  great  caution.  Many  of  the  features  of  the  model  have  been  significantly  impacted  by 
recent  changes  in  the  acquisition  process.  For  instance,  the  MANPRINT  program  has  had 
a  considerable  amount  of  influence  on  the  treatment  of  manpower,  training,  human  factors 
and  safety  issues.'  Some  caution  must  also  be  used  in  applying  DARCOM/TRADOC 
Pamphlet  70-2. 

2.3.2  The  Traditional  Materiel  Acquisition  Process 

2.3.2.1  Overview  (AR  70-1. Pam  70-2) 

The  Traditional  acquisition  process  is  divided  into  five  phases.  They  are  the  (1) 
Program  Initiation,  (2)  Concept  Exploration,  (3)  Demonstration  and  Validation,  (4)  Full 
Scale  Development  (FSD),  and  (5)  Production  and  Deployment  Phases.  Production  and 
Deployment  may  be  divided  into  Low  Rate  Initial  Production  and  a  full  Production  and 
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Deployment  Phase.  Each  succeeding  phase  takes  a  materiel  requirement  from  a  broad 
concept  to  successively  more  precisely  specified,  producible  and  supportable  materiel 
solution  to  the  Army’s  operational  needs.  The  traditional  process  typically  consumes  about 
1 1  to  15  years  from  program  initiation  to  fielding. 

OWL  considerations  and  trade-offs  should  be  made  early  in  this  life  cycle.  A 
commonly  accepted  rule  of  thumb  holds  that  70  percent  of  system  life  cycle  costs  are  set  at 
or  before  the  system  proceeds  into  the  Demonstration  and  Validation  Phase  ( Advanced 
Development ).  Paralleling  this  is  the  observation  of  a  similar  accelerating  decline  in  design 
flexibility.  Figure  2.3.2-1  illustrates  the  traditional  process  and  where  OWL  evaluations 
and  predictions  may  be  used  most  appropriately.  OWL  consideration  must  be  made  in  the 
early  phases  of  development  because  the  costs  of  modifying  design  later  accelerate  and 
options  similarly  decline  as  the  system  proceeds  further  into  the  MAP. 

2.3.2.2  Program  Tnitiation  (AR  70-1 .  AR  71-9.  PAM  70-2^ 

Activities  which  result  in  program  initiation  activities  can  be  divided  into  two  broad 
categories.  These  are  mission  and  operations  oriented  activities  and  science  and  technology 
oriented  activities.  Mission  and  operations  oriented  activities  include  threat  assessments, 
conduct  of  Mission  Area  Analyses  (MAA),  development  of  long  range  plans  such  as  the 
Battlefield  Development  Plan  (BDP)  and  the  Mission  Area  Materiel  Plan  (MAMP),  and 
preparation  of  the  DA  Long  Range  Research  Development  and  Acquisition  Plan 
(LDRRDP). 
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OWL  IN  THE  NORMAL  MATERIEL  LIFE  CYCLE 


Figure  2.3.2-1  OWL  Considerations  in  the  Traditional  MAP 


Science  and  technology  oriented  activities  include  basic  and  exploratory  research 
and  other  technical  base  activities  which  lead  to  new  or  improved  technological  capabilities. 
The  operations  and  mission  oriented  activities  may  lead  to  the  development  and  submission 
of  a  Justification  for  Major  Systems  New  Start  (JMSNS),  upon  which  a  major  DoD 
program  may  be  based.  An  Operational  and  Organizational  Plan  (O&O  Plan),  is  the  basis 
for  initiating  a  designated  acquisition  program  (DAP)  at  the  Department  of  Army  level  or 
non-major  development  program  at  levels  below  DA.  The  O&O  Plan  is  also  a  part  of  the 
JMSNS.  OWL  considerations  have  potential  contributions  to  all  these  activities.  For 
example,  basic  and  exploratory  research,  and  technical  base  programs  may  directly  address 
OWL  issues.  Mission  Area  Analyses  (MAAs)  may  conclude  that  operational  work  load 
deficiencies  may  constitute  future  program  drivers.  The  O&O  plan  may  be  constrained  by 
OWL  considerations. 

The  mission  oriented  activities  include  the  thirteen  specific  MAAs  which  have  been 
conducted  and  are  periodically  updated  by  the  US  Army  Training  and  Doctrine  Command 
(TRADOC).  These  analyses  provide  an  in-depth  examination  of  the  Army’s  ability  to 
perform  its  fundamental  combat  missions  (e.g.  Close  Combat,  Fire  Support,  Air  Defense, 
NBC,  etc.).  OWL  considerations  may  limit  specific  operational  capabilities  addressed 
within  given  mission  areas.  These  would  be  expected  to  be  statements  of  OWL  limitations 
based  on  evaluation  of  existing  capability. 

The  BDP  and  MAMP  are  broad  examinations  of  the  Army's  operational  and 
materiel  capabilities.  They  are  based  on  the  MAA  and  prioritize  future  program  efforts 
based  on  the  DA  LRRDAP.  If  a  materiel  solution  is  considered  appropriate  to  address  a 
mission  area  deficiency,  that  solution  is  supported  in  an  O&O  Plan.  The  O&O  plan  may 
be  prepared  as  a  stand-alone  document  or  may  be  an  attachment  to  the  JMSNS.  OWL 
considerations  play  an  important  role  in  developing  the  O&O  plan.  The  O&O  plan 
(Appendix  C,  AR  71-9)  describes  how  a  system  will  be  integrated  into  the  force  structure, 
and  deployed,  operated  and  supported  during  both  peace  time  and  war  time.  It  ultimately 
supports  the  preparation  of  detailed  integrated  logistic  support  planning,  basis  of  issue 
planning,  and  broad  personnel  planning.  MANPRINT  considerations  and  personnel 
impacts  are  a  specific  section  within  the  O&O  Plan.  Assessment  of  these  personnel  impacts 
need  to  be  supported  by  OWL  predictions  and  evaluations  for  similar  existing  systems. 

Technical  base  activities  include  basic  research  and  exploratory  development  which 
address  questions  which  impact  on  future  operational  capabilities.  OWL  issues  need  to  be 


addressed  within  the  framework  of  these  research  programs.  Technical  base  activities  are 
prioritized  and  funded  in  accordance  with  Science  and  Technology  Objectives  (STOs) 
which  are  established  and  prioritized  in  the  HQ  DA  LRRDAP.  The  STOs  are  based  on 
operational  deficiencies  noted  in  MAAs. 

Program  initiation  results  when  it  is  recognized  that  an  operational  deficiency  exists 
and  it  is  likely  that  a  materiel  solution  will  be  effective.  Typically,  a  Special  Task  Force 
(STF),  Special  Study  Group  (SSG)  or  acquisition  team  is  formed  on  approval  of  the  O&O 
Plan  or  JMSNS.  Secretary  of  Defense  guidance  may  be  established  for  major  programs  in 
a  Program  Decision  Memorandum  (PDM).  The  PDM  provides  broad  program  guidance 
from  the  DoD  level  for  systems  for  which  a  JMSNS  is  required.  The  STF/SSG  or 
Acquisition  Team  carry  the  program  responsibilities  until  the  program  enters  the  Concept 
Exploration  Phase. 

2.3.2.3  Concept  Exploration  tAR  70-1.  AR  71-9.  AMC  Reg  70-52.  Pam  70-2. 

AR  602-2') 

During  concept  exploration,  technical  alternatives  to  meeting  the  stated  material  need 
and  supporting  that  system  are  identified  and  explored.  The  competing  alternatives  include 
incorporation  of  new  technology,  adoption  of  non-development  items  (NDI),  and  product 
improvement  of  existing  hardware.  Market  surveys  are  conducted,  bread  board  and 
experimental  prototypes  are  developed  and  tested,  and  information  regarding  development 
risks  and  program  alternatives  are  developed  and  assessed.  Final  system  concepts  are 
ultimately  selected  and  plans  addressing  training,  logistics,  and  future  testing  are 
developed.  An  Acquisition  Strategy  (AS)  is  developed  which  guides  the  conduct  of  the 
future  program. 

The  concept  exploration  phase  may  be  the  most  critical  for  the  application  of  OWL 
concepts.  Trade  offs  addressing  technical  approaches  and  training  and  support  approaches 
will  impact  on  virtually  all  future  related  development  activities.  OWL  issues  addressed  in 
the  trade  off  determination  (TOD),  trade  off  analysis  (TO A)  and  development  of  the  best 
technical  approach  (BTA)  will  become  embedded  in  the  program  for  its  entire  life.  OWL 
considerations  will  also  impact  on  other  studies  and  analysis  conducted  during  this  phase. 
System  engineering  activities  commence  and  serve  as  a  significant  tool  for  integrating  OWL 
requirements  into  the  system.  Update  of  the  O&O  plan  and  development  of  cost  and 
operational  effectiveness  analysis,  development  of  base  line  cost  estimates  and  independent 
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parametric  cost  estimates  will  all  be  impacted  by  OWL  considerations.  Typical  concept 
formulation  activities  are  described  briefly  with  comments  on  their  relationship  to  operator 
workload  issues.  These  activities  include  : 

•  O&O  Plan  —  The  previously  developed  O&O  Plan  is  revised. 
Requirements  stated  within  the  O&O  Plan  should  be  based  on  an 
understanding  of  potential  OWL  impacts.  OWL  assessments  based  on 
similar  systems  may  be  employed  in  developing  the  plan. 

•  Concept  Formulation  Package  —  The  CFP  consists  of  the  Trade  Off 
Determination  (TOD),  the  Trade  Off  Analysis  (TOA),  the  Best  Technical 
Approach  (BTA),  and  the  Cost  and  Operational  Effectiveness  Analysis 
(COEA).  These  analyses  are  packaged  as  a  part  of  the  CFP  under  a 
cover  letter  which  summarizes  essential  system  features  to  include 
MANPRINT  requirements  (Appendix  E,  AR  71-9),  and  performance 
characteristics. 

•  Trade  Off  Determination  -  The  TOD  establishes  viable  trade  offs  for  the 
suggested  approach  in  pursuing  the  development  program.  It  includes 
life  cycle  costing  and  scheduling  information.  ILS  and  MANPRINT 
requirements  are  included  in  issues  which  must  be  addressed  within  the 
TOD.  OWL  predictions  regarding  one  or  more  trade  off  alternatives 
may  be  required. 

•  Trade  Off  Analysis  -  TOA  is  prepared  jointly  by  the  combat  developer 
and  material  developer.  TOA  serves  to  select  the  best  technical  approach 
based  on  the  alternatives  presented  in  the  trade  off  determination. 

Again,  MANPRINT  and  ILS  requirements  are  essential  trade  offs. 

OWL  predictions  which  consider  system  impacts  resulting  from  OWT^ 
considerations  may  be  important  in  completing  the  TOA. 

•  Best  Technical  Approach  —  BTA  is  prepared  jointly  by  the  materiel  and 
combat  developer.  It  documents  the  BTA  to  include  ILS  and 
MANPRINT  concepts  based  on  the  results  of  the  TOD  and  TOA.  The 
results  of  previous  OWL  predictions  are  considered  in  formalizing  the 
BTA. 

•  Cost  and  Operational  Effectiveness  Analysis  —  The  COEA  is  prepared 
by  the  combat  developer.  It  examines  the  cost  and  operational 
effectiveness  of  competing  alternatives.  All  important  systems  aspects 
should  be  considered  in  addressing  COEA  to  include  OWL.  OWL 
predictions  and  the  results  of  OWL  evaluations  on  bread  board  and 
brass  board  prototypes  can  provide  important  information. 

•  Bread  Board/Brass  Board  Tests  —  Bread  board  and  brass  board 
prototypes  are  normally  fabricated  and  tested  during  the  course  of 
concept  evaluation  .  Early  OWL  evaluations  against  these  prototypes 
will  provide  an  important  baseline  for  future  OWL  predictions  and  the 
development  of  future  OWL  evaluation  methodology. 

•  Acquisition  Strategy  —  The  AS  is  the  basic  program  plan  for  the 
development  program.  It  is  prepared  by  the  material  developer  in 
coordination  with  the  other  members  of  the  acquisition  team.  It 
provides  guidance  on  tailoring  the  acquisition  process  for  the  specific 
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development  and  highlights  potential  risks  and  plans  to  reduce  or 
eliminate  risks.  MANPRINT  is  specifically  addressed  in  the  AS  (AR 
70-1,  paragraph  5-2b).  OWL  is  an  issue  to  be  considered  under 
MAl^RINT  analyses. 

*  System  Safety  Program  Plan  —  Safety  and  health  hazard  assessment 
issues  have  been  addressed  and  identified  for  further  resolution  during 
the  development  process.  These  measures  are  addressed  in  the  SSPP. 
Potential  OWL  issues  may  result  from  predictive  analyses.  OWL 
evaluation  may  result  from  bread  board  and  brass  board  prototype  tests. 
Considerations  may  include  operational  constraints,  training 
requirements  and  restrictions. 

*  Integrated  Logistics  Support  —  Integrated  logistics  support  planning  is 
initiated.  This  includes  issues  such  as  maintenance  planning,  manpower 
and  personnel  requirements,  training  operating  personnel,  and 
requirements  for  training  devices.  ILSPs  prepared  during  the  concept 
formulations  phase  may  include  the  results  of  investigations  based  on 
performance  data  from  deployed  systems. 

*  Training  Plans  —  Training  plans  developed  during  the  concept 
exploration  phase  address  alternative  training  concepts.  They  are 
designed  to  highlight  critical  training  areas  for  consideration  during  the 
balance  of  the  development  process,  therefore  contributing  to  the 
ultimate  system  availability,  maintainability,  and  operational  capability. 
Human  factors  considerations  in  general,  and  OWL  considerations  in 
particular  are  important  training  planning  drivers.  OWL  predictions 
based  on  OWL  evaluations  of  early  bread  board  and  brass  board,  and 
similar  systems  currently  in  the  field  are  important  sources  of  data  for 
the  preparation  of  training  plans.  These  plans  include  assessment  of 
appropriate  training  methods,  media,  training  devices,  skill 
qualification,  evaluation  procedures,  and  the  need  for  training 
simulations. 

*  System  MANPRINT  Management  Plan  —  The  SMMP  is  the 
management  document  that  describes  MANPRINT  concerns  and  tasks 
and  analyses  that  need  to  be  conducted  during  the  acquisition  to  ensure 
consideration  of  manpower,  personnel,  training,  human  factors 
engineering,  system  safety,  and  health  hazard  assessment.  These  issues 
are  addressed  and  resolved  or  identified  for  resolution  during  following 
phases.  OWL  assessments  of  existing  hardware  may  be  useful  in 
comparative  analyses.  The  SMMP  is  initiated  and  maintained  by  the 
combat  or  training  developer  and  is  updated  throughout  the  acquisition 
process  (AR  602-2).  (See  Section  2.4.3  for  further  discussion  of  the 
SMMP). 

*  Test  and  Evaluation  Master  Plan  —  Test  and  Evaluation  Master  Plan 
(TEMP)  developed  during  the  concept  formulation  provides  the 
foundation  for  development  and  operational  testing  throughout  the 
balance  of  the  program.  The  TEMP  provides  the  frame  work  for 
showcasing  critical  development  and  operational  issues  which  need  to 
be  addressed  during  future  testing.  Operational  issues  which  may  be 
addressed  during  development  testing,  and  the  expansion  of  the 
development  test  data  base  during  operational  testing  are  important 
issues  for  consideration  within  the  TEMP.  Critical  OWL  issues  (such 
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as  those  identified  in  the  SMMP)  are  highlighted  in  conjunction  with 
other  MANPRINT  and  human  factors  related  test  issues. 

•  System  Engineering  Management  Plan  --  The  System  Engineering 
Management  Plan  (SEMP)  is  prepared  in  accordance  with  AMC 
Regulation  70-52.  The  SEMP  will  provide  a  framework  for  the 
integration  of  system  requirements  including  MANPRINT  and  OWL 
considerations  Application  of  sound  system  engineering  principles, 
based  on  the  SEMP,  will  be  a  required  feature  throughout  die  life  of  the 
program. 

•  System  Concept  Paper  -  The  SCP  summarizes  the  activity  of  the 
Concept  Exploration  Phase  and  is  the  basis  for  obtaining  approval  to 
proceed  to  the  next  phase  of  development.  Although  the  SCP  is  very 
brief,  not  exceeding  12  pages,  it  is  a  key  document  in  the  Materiel 
Acquisition  Decision  Process  (MADP).  OWL  issues  would  not 
normally  be  expected  to  be  highlighted  within  the  SCP.  However, 
critical  OWL  issues  may  be  highlighted  under  the  MANPRINT  or  HFE 
paragraphs  in  the  Acquisition  Strategy  (Annex  F). 

Programs  in  the  Concept  Explanation  phase  may  be  under  a  program  manager,  may 
continue  to  be  managed  by  the  SSG/STF,  or  may  be  managed  by  an  acquisition  team 
appointed  by  the  developing  agency.  TRADOC  will  typically  follow  the  program  through 
an  appropriate  TRADOC  Systems  Manager  (TSM)  with  input  from  the  Combat 
Development  and  Training  Development  Directorates  of  the  proponent  school. 

2.3.2.4  Demonstration  and  Validation  Phase  (  AR  70-1.  AR  70-10.  AMC  Reg  70- 
52.  Pam  70-2  .  AR  602-21 


The  Demonstration  and  Validation  Phase  is  conducted  to  verify  preliminary  designs 
and  engineering,  and  to  accomplish  the  necessary  planning  and  trade  offs  to  reduce  risks  in 
future  development  and  fielding.  ILS  and  MANPRINT  issues  are  addressed  early-on  (AR 
70-1,  paragraph  3-5a).  Emphasis  includes  conducting  trade  offs  among  system 
characteristics,  manpower,  personnel,  training,  and  support  concepts.  Other  important 
tasks  include  preparation  of  the  Required  Operational  Capability  documentation  and 
training  device  requirements.  User  participation  is  important  during  testing  in  order  to 
prove  out  the  O&O  concept.  OWL  concepts  and  considerations  are  applied  throughout  this 
phase  of  development.  It  is  essential  to  thoroughly  apply  OWL  evaluation  capabilities 
during  this  phase  in  order  to  insure  that  OWL  related  trade  offs  are  thoroughly  understood. 
The  cost  of  modifying  designs  for  the  sake  of  OWL  enhancements  becomes  progressively 
more  expensive  as  the  program  proceeds  to  full  scale  development. 


2-10 


The  demonstration  and  validation  phase  features  procurement  of  advanced 
development  prototypes  for  testing  during  development  and  operational  tests  .  Frequently, 
competing  prototypes  will  be  developed,  fabricated,  and  tested.  The  advanced 
development  prototypes  are  designed  based  on  functional  requirements  and  are  consistent 
with  the  Best  Technical  Approach  articulated  in  the  Concept  Formulation  Package. 
Technical  and  operational  tests  provide  an  excellent  fomm  for  validating  OWL  predictions 
made  during  the  Concept  Formulation  Phase.  It  is  important  to  ensure  that  OWL  issues  are 
addressed  during  these  tests  in  order  to  develop  a  clear  understanding  of  OWL  issues 
which  need  be  addressed  in  future  development  and  production.  Typical  advanced 
development  activities  are  described  briefly  with  comments  on  their  relationship  to  operator 
workload  issues.  These  activities  include: 

•  Advance  Development  Contract  --  Advanced  development  prototypes 
are  based  on  the  preferred  alternatives  studied  during  concept 
formulation.  Frequently,  two  or  more  competing  prototypes  will  be 
procured  for  development  and  operational  test  and  evaluation.  OWL 
specialists  should  be  involved  in  developing  the  technical  data  package 
for  the  Advanced  Development  Contracts  and  in  evaluating  the  resulting 
proposals.  The  advanced  development  prototypes  provide  an  efficient 
form  for  addressing  the  impacts  and  potential  solutions  to  OWL 
problems  in  fielded  items.  0\^  related  designs  features,  OWL  studies, 
and  OWL  evaluations  may  be  all  addressed  within  the  frame  work  of  the 
Advanced  Development  Contracts. 

•  Technical  and  Operational  Tests  -  Technical  and  Operational  tests  are 
conducted  on  advanced  development  prototypes  and  address  critical 
issues  established  in  the  TEMP.  Both  government  and  contractor 
developed  test  data  is  used  to  address  these  issues.  OWL  specialists 
participate  in  or  closely  follow  the  conduct  of  these  tests.  In  those  cases 
where  specific  OWL  issues  are  being  evaluated  within  those  test 
programs,  OWL  specialists  will  develop  test  procedures,  collect  or 
supervise  collection  of  OWL  related  data,  and  report  and  evaluate  the 
results  of  those  tests.  Frequently  the  developer  and  user  address 
respective  critical  issues  during  the  same  test  procedures.  Close 
coordination  with  all  individuals  on  the  test  team  will  enhance  the 
efficient  and  effective  collection  of  OWL  related  data.  Established 
OWL  evaluation  methodology,  or  specific  methodology  developed 
during  previous  development  phases,  will  be  applied  during  technical 
and  operational  tests. 

•  ILSP  Updates  —  The  ILSP,  prepared  during  the  Concept  Formulation 
Phase,  receives  a  complete  update  during  demonstration  and  validation. 

This  update  is  based  on  the  results  of  testing.  The  updated  ILSP 
examines  support  planning  concepts,  establishes  a  baseline  support 
concept,  identifies  support  parameters,  and  examines  potential 
supportability  problems.  Logistic  systems  and  resource  constraints,  and 
recommended  reliability  and  maintainability  parameters,  are  formulated. 


•  Training  Support  Planning  Update  -  The  plans  for  training  support  are 
updated  by  the  trainer  and  the  material  developer  in  coorcUnation  with 
the  combat  developer  and  logistician.  This  update  is  done  in 
conjunction  with  preparation  of  the  tentative  qualitative  and  quantitative 
personnel  requirements  information  (TQQPRI)  which  has  significant 
OWL  impact  The  plans  for  training  support  will  include  plans  for  new 
equipment  training.  Training  support  plans  establish  the  baseline  for 
future  training  during  full  scale  development,  initial  production  and 
fielding,  and  subsequent  support  of  the  system  in  the  field.  Potential  or 
identified  OWL  problems  may  be  addressed  through  training. 

•  Tentative  Qualitative  and  Quantitative  Personnel  Requirements  Inventory 
“  The  TQQPRI  is  prepared  by  the  material  developer.  The  TQQPRI  is 
based  on  human  factors  studies,  logistics  support  analysis,  development 
of  a  system  training  strategy,  and  behavioral  research.  It  describes 
personnel  duties,  tasks  down  to  work  units,  performance  standards,  the 
basis  for  manpower  authorization  factors,  recommended  Military 
Occupational  Specialties  (MOS)  to  include  skill  levels,  and 
recommended  organizations.  The  TQQPRI,  in  conjunction  with  the 
Integrated  Logistics  Support  Plans,  are  essential  in  developing  hardware 
basis  of  issue  plans  (BOIP),  training  device  requirements,  and  other 
training  in  support  issues.  The  TQQPRI  provides  the  most  current 
information  concerning  numbers  and  qualifications  of  personnel 
required  for  employment  support  and  maintenance  of  the  system. 

•  Tentative  Basis  of  Issue  Plan  —  The  TBOIP  is  prepared  by  the  combat 
developer  in  consideration  of  studies  conducted  by  the  combat  developer 
and  the  TQQPRI.  It  serves  as  the  basis  for  future  development  of  how 
the  system  will  be  distributed  and  supported  within  the  Army.  The 
TBOIP  is  developed  on  the  basis  of  the  best  available  information. 

•  Training  Device  Requirement  --  TDR  requirements  are  developed  by  the 
trainer  in  conjunction  with  development  of  training  support  plans  and 
the  TQQPRI.  Training  devices  used  during  testing  and  future  tests  may 
also  be  useful  tools  in  investigating  OWL  issues. 

•  Required  Operational  Capability  --  The  ROC  establishes  essential 
operational,  RAM,  technical,  personnel  and  manpower,  training,  safety 
and  health,  human  factors  engineering,  logistical,  and  cost  information. 
It  is  used  as  a  basis  for  proceeding  with  full-scale  development.  Letter 
Requirements  (LRs)  are  prepared  for  similar  purpose  for  low  dollar 
value  items.  MANPRINT  issues  are  addressed  in  a  specific  section 
(Para^aph  8)  of  the  ROC.  Operational  trainability  and  the  technical 
feasibility  of  the  proposed  system  is  also  addressed.  OWL  inputs  to  a 
ROC  (or  a  LR)  are  based  on  the  results  of  previous  OWL  evduations 
and  current  OWL  predictions.  The  most  appropriate  location  for  OWL 
inputs  would  be  in  the  MANPRINT  section. 

•  Safety  and  Health  Hazard  Assessment  —  The  Safety  and  Health  Hazard 
Assessment  is  developed  based  on  test  results  and  other  demonstration 
and  validation  phase  activities.  Results  of  OWL  evaluations  conducted 
during  tests  and  potential  OWL  problems  may  be  used  in  updating  the 
Safety  and  Health  Hazard  Assessment. 
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Human  Factors  Engineering  Analysis  —  The  HFEA  is  conducted  to 
identify  any  human  factors  problems  associated  with  the  system. 
Suitability  of  the  system  to  proeeed  to  the  Full  Scale  Development  Phase 
is  established.  Human  factors  issues,  including  OWL  for  resolution  are 
highlighted. 

System  MANPRINT  Man^ement  Plan  --  The  SMMP  is  updated  based 
on  demonstration  and  validation  phase  results.  Plans  for  future 
MANPRINT  related  activity  are  incorporated. 

Technical  Data  Package  -  The  technical  data  package  (TDP)  for  a  full- 
scale  engineering  development  is  a  detailed  specification  for  engineering 
development  prototypes.  The  specifications  must  be  in  sufficient  detail 
to  insure  delivery  of  hardware  and  software  which  is  characteristic  of 
that  which  may  be  delivered  from  production  processes.  Such  detailed 
specifications  may  be  useful  in  determining  hardware  design 
charaeteristics  that  impact  OWL. 

Acquisition  Strategy  Update  —  The  AS  developed  during  the  Concept  - 
Formulation  Phase  is  updated  to  support  full-scale  engineering 
development  and  subsequent  production  and  fielding.  MANPRINT 
issues,  examined  during  the  development  of  the  original  AS  are  re¬ 
examined,  updated,  and  expanded.  The  results  of  OWL  evaluations 
conducted  during  the  validation  phase  and  current  OWL  predictions  are 
used  as  a  basis  for  preparing  OWL  related  inputs  to  the  AS. 

Test  and  Evaluation  Master  Plan  Update  —  The  TEMP  is  updated  to 
reflect  test  and  evaluation  requirements  for  full-scale  development,  and 
production  and  fielding.  0>AT.  critical  issues  to  be  addressed  during 
technieal  and  operational  testing  and  subsequent  production  related 
testing  must  be  articulated  in  this  TEMP  update.  OWL  critical  issues  are 
based  on  OWL  evaluation  results  from  the  tests  and  predictions  made 
during  the  validation  phase. 

Cost  and  Effectiveness  Analysis  —  COEA  is  updated  based  on  the 
results  of  tests  and  studies  conducted  during  the 
demonstration/validation  phase.  The  resolution  of  OWL  issues  may 
have  substantial  impacts  on  COEA  results.  The  results  of  OWL 
evaluations  conducted  during  the  validation  phase  and  current  OWL 
predictions  are  used  as  a  basis  for  the  COEA  update. 

System  Engineering  Management  Plan  —  System  Engineering  activities 
continue  in  order  to  insure  integration  of  required  system  features  on  a 
total  system  basis.  The  SEMP  is  updated  to  support  full-scale 
development.  These  activities  are  a  major  tool  for  integrating 
MANPRINT  in  general  and  OWL  considerations  into  the  development 
program. 

Decision  Coordinating  Paper  --  The  DCP  summarizes  the  results  of  the 
demonstration/validation  phase,  and  provides  a  recommendation  for 
proceeding  with  development  of  the  system.  It  is  a  key  document  in  the 
MADP  for  obtaining  a  decision  to  proceed  to  FSD.  The  DCP  is  not  to 
exceed  eighteen  pages,  excluding  six  annexes.  The  DCP  includes  a 
description  of  the  alternatives  considered  during  the 
demonstration/validation  phase,  and  a  description  of  the  selected 
alternative  to  include  the  operational  concept  for  the  selected  alternative. 


2-13 


Sustainability  and  economy  of  manpower  are  issues  to  be  included  in 
that  discussion.  The  DCP  also  includes  technological  risks  for  the 
selected  alternative  and  how  those  risks  have  been  resolved  in  the 
demonstration/validation  phase.  OWL  inputs  to  the  DCP  are  based  on 
OWL  evaluations  conducted  during  the  demonstration/validation  phase 
and  current  OWL  predictions.  The  DCP  includes  the  AS  (Annex  F)  and 
its  attendant  MANPRINT/Human  Factors  section. 

The  Demonstration  and  Validation  Phase  is  managed  by  a  Project  Manager,  or  by 
an  acquisition  team  appointed  from  within  the  developing  agency.  DoD  major  programs 
and  DAP  will  be  conducted  under  a  DOD  or  DA  chartered  Program  Manager.  TRADOC 
will  typically  continue  to  follow  the  major  systems  through  an  appropriate  TRADOC 
Systems  Manager  (TSM)  with  input  from  the  Combat  Developments  Directorate  of  the 
proponent  school.  IPR  programs  which  are  not  under  project  or  product  managers  will  be 
managed  by  an  acquisition  team  appointed  by  the  developing  agency. 

13.1.5  Full-Scale  Development  (AR  70-1.  AR  70-10.  AMC  Reg  70-52.  Pam70-2  1 

Full  Scale  Development  provides  an  opportunity  to  completely  evaluate  the  system 
in  the  form  expected  to  be  fielded.  Systems  which  successfully  complete  full  scale 
development  are  type  classified  as  standard  and  are  procured  for  issue  to  the  field.  The 
total  system  is  normally  prototyped,  tested  and  evaluated,  to  include  all  support  systems 
and  software.  These  include  simulators,  training  devices,  computer  equipment,  training, 
and  maintenance  manuals.  OWL  evaluation  expertise  is  required  to  ensure  that  OWL  issues 
have  been  addressed  in  the  full  scale  development  prototypes. 

There  is  limited  opportunity  to  make  changes  in  the  system  which  will  enhance 
OWL  performance  during  this  phase  of  development.  Essential  changes  may  be  considered 
if  there  is  a  significant  OWL  problem.  However,  changes  to  any  part  of  the  system  tend  to 
cascade  throughout  the  system,  to  include  software  and  manuals.  As  a  result,  changes  are 
expensive  and  time  consuming.  Most  changes  not  demonstrated  as  essential  for  meeting 
system  requirements  would  be  considered  during  product  improvement  programs 
conducted  after  initial  production  and  fielding.  OWL  inputs  which  have  an  impact  on 
system  design,  therefore,  are  most  effectively  made  prior  to  entry  into  this  phase.  Typical 
full  scale  development  activities  are  described  briefly  with  comments  on  their  relationship 
to  workload  issues.  These  activities  include: 

•  Full  Scale  Development  Contract  --  The  full  scale  development  contract 
is  solicited  on  the  basis  of  the  technical  data  package  developed  during 
the  validation  phase.  Full  scale  development  prototypes  are  normally 
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typical  in  form  and  function  of  the  expected  production  prototype.  The 
contract  also  calls  for  evaluation  and  delivery  of  a  complete  suite  of 
support  equipment,  to  include  test  equipment,  training  devices, 
software,  manuals,  etc.,  as  well  as  preparation  of  a  technical  data 
package  for  production.  OWL  related  criteria  should  be  one  of  the  bases 
for  ev^uation  of  full  scale  development  proposals. 

•  Technical  and  Operational  Testing  —  The  complete  system  is  tested  and 
evaluated  against  criteria  documented  in  the  ROC  or  LR  and  developed 
in  the  TEMP.  The  Materiel  Developer,  and  his  contractor  and  technical 
tester  (normally  the  US  Army  Test  and  Evaluation  Command-  TECOM) 
prepare  and  execute  technical  testing  in  accordance  with  the  TEMP. 
The  operational  tester,  either  the  Army  Operational  Test  and  Evaluation 
Agency  (OTEA)  (  for  major  systems)  or  a  designated  Training  and 
Doctrine  Command  (TRADOC)  school,  address  operational/user  issues 
as  specified  in  the  TEMP.  The  resulting  test  reports  are  used  to  prepare 
technical  and  operational  Independent  Evaluations.  Specialists  develop 
tests,  or  make  input  into  more  general  tests,  in  order  to  address  OWL  - 
issues  highlighted  in  the  TEMP.  OWL  criteria  which  serve  as  a  basis 
for  accepting  hardware  or  support  equipment  and  software  may  be 
addressed  in  contractor  testing  for  subsequent  review  by  the 
government.  OWL  evaluation  techniques  are  integrated  as  required  into 
all  testing. 

•  SMMP  Update  —  The  SMMP  is  updated  throughout  the  cycle  to  reflect 
analyses  performed,  questions  and  concerns  addressed,  new 
MANPRINT  concerns  that  have  been  raised,  as  well  as  maintain  an 
audit  trail  of  all  decisions  and  work  that  has  been  done  in  support  of  the 
SMMP. 

•  ILSP  Update  —  The  updated  ILSP  is  based  on  the  ILSP  prepared  during 
the  validation  phase.  The  plan  is  expanded  to  address  support  of  the 
system  as  it  is  introduced  into  the  field.  The  ILSP  is  updated  in  concert 
with  updates  to  the  QQPRI,  BOIP,  incorporation  into  tables  of 
organization  and  equipment  (TOE),  preparation  of  the  Materiel  Fielding 
Plan,  and  other  ILS  oriented  actions.  Test  results  and  specific  ILS 
studies  are  also  used  as  a  basis  for  the  update.  The  results  of  OWL 
evaluations  addressed  during  testing  and  other  OWL  studies  provide 
OWL  input. 

•  TEMP  Update  —  The  TEMP  is  updated  to  support  testing  required 
during  the  production  and  deployment  phase.  Testing  required  to 
demonstrate  that  major  deficiencies  noted  during  technical  and 
operational  tests  are  corrected  is  described.  Production  validation  tests 
and  follow-on  operational  test  and  evaluation  requirements  are 
established.  Requirements  for  first  article  and  initial  production  tests  are 
also  established. 

•  AS  Update  —  The  Acquisition  Strategy  is  updated  to  emphasize 
production  and  deployment  requirements.  Critical  OWL  issues  which 
remain  to  be  addressed  during  the  production  and  deployment  phase 
may  be  presented  as  a  MANPRINT  consideration. 

•  Requirement  Documents  Revisions  —  Revisions  to  the  requirements 
document  developed  during  the  Demonstration  and  Validation  Phase 


(ROC,  LR,  TDR)  must  be  considered  to  support  production  and 
deployment.  Technical  and  operational  characteristics  may  require 
revisions  for  a  number  of  reasons  including  changes  to  the  established 
threat.  Any  changes  to  the  established  requirement  must  be  approved  by 
both  AMC  and  T^DOC  and,  in  conjunction  with  the  MADP,  DA. 

•  Decision  Coordinating  Paper  —  The  DCP  is  prepared  to  support  the 
production  and  deployment  decision.  It  summarizes  the  results  of  full 
scale  development  and  document  recommendations  for  production  and 
deployment.  It  includes  a  full  description  of  the  commitments  the  Army 
is  making  in  proceeding  with  production,  to  include  budget 
requirements  and  future  support  requirements.  TTie  revised  Acquisition 
Strategy,  with  its  required  MANPRINT  and  human  factors  sections,  is 
an  annex  to  the  DCP. 

Typically,  the  same  type  of  program  management  method  is  continued  from  that 
used  in  the  Demonstration  and  Validation  Phase.  Development  programs  may  be  managed 
by  a  program  or  project  manager,  or  by  an  acquisition  management  team  appointed  by  the 
developing  agency.  TRADOC  management  is  also  normally  continued  with  a  TSM 
supported  by  combat  development  and  training  development  elements.  Programs  which 
are  project  managed  normally  continue  with  that  form  of  management,  at  least  until  the 
system  is  successfully  fielded. 

2.3.2.5  Production  and  Deployment  (AR  70-1.  Pam  70-21 

During  production  and  deployment  operational  quantities  of  the  system  and  required 
support  equipment  is  procured,  personnel  and  operational  units  are  trained,  and  the  logistic 
support  system  is  implemented  to  support  the  system  in  the  field.  If  directed  at  the 
Milestone  III  decision  point.  Low  Rate  Initial  Production  may  be  implemented  to  address 
additional  testing,  production  engineering  or  production  base  issues. 

Follow-on  Operational  Test  and  Evaluation  (FOTE)  may  be  conducted  during  the 
production  and  deployment  phase,  if  required,  to  address  a  spectrum  of  issues,  including 
those  with  manpower  and  training  impacts.  FOTE  has  the  potential  to  serve  as  relatively 
well  controlled  forum  for  developing  OWL  data  upon  which  to  base  product  improvement 
or  new  equipment  requirement  issues. 
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2.3.3  The  Army  Streamlined  Acquisition  Process  ( AR  70-1 ) 


2.3.3. 1  Overview 

The  Army  Streamlined  Acquisition  Process  (ASAP)  compresses  the  standard 
acquisition  cycle  from  over  eleven  years  to  seven  years  or  less.  The  objective  of  the 
streamlined  process  is  to  achieve  operational  capability  within  the  minimum  practical 
amount  of  time,  for  low  risk  development  programs.  Programs  which  need  more  detailed 
and  deliberate  development  processes,  generally  characterized  as  higher  risk  development 
programs,  may  still  follow  all  or  portions  of  the  "traditional"  acquisition  process. 

The  streamlined  process  is  characterized  by  four  distinct  phases  as  seen  in  Figure 

2.3.3-  1.  They  are  1)  Requirements  and  Technical  Base  Activities,  2)  Proof  of  Principle,  3) 
Development-Production  Prove  Out,  and  4)  Production  and  Deployment.  Proof  of 
Principle  is  followed  by  a  go/no-go  decision  (Milestone  I/II)  to  proceed  into  the 
development/prove  out  phase.  The  focus  of  the  streamlined  process  is  in  the 
development/prove  out  phase.  Activities  typical  of  traditional  full-scale  development  and 
early  production/deployment  are  conducted  as  expeditiously  as  practical. 
Development/prove  out  is  followed  by  a  Milestone  HI  decision  point  in  order  to  provide 
approval  to  enter  full  production  and  deployment.  The  objective  of  the  streamlined  process 
is  to  achieve  a  Milestone  III  decision  in  fours  years  or  less  from  demonstration  of  proof  of 
principle.  Overlaid  on  the  streamlined  acquisition  procedure  are  provisions  for  preplanned 
product  improvements  (P3I)  throughout  the  life  of  the  system.  New  or  improved 
technology  is  "inserted"  at  appropriate  points  throughout  the  life  cycle  as  the  threat  or  the 
technology  changes.  These  insertion  points  may  be  during  the  developmen^prove  out 
stage,  as  well  as  during  the  production  and  deployment  phase  or  later  in  the  cycle. 

Documentation  prepared  during  the  conduct  of  programs  under  the  streamlined 
acquisition  process  is  similar  to  that  prepared  under  the  traditional  process.  OWL 
considerations  for  analysis  and  the  development  of  design  approaches  and  documentation 
are  identical  in  comparison  to  those  conducted  during  the  traditional  process.  Successive 
iterations  of  documentation,  as  the  system  proceeds  through  the  stages  of  development 
under  the  ttaditional  process,  are  eliminated.  OWL  considerations  for  the  development  of 
specific  documentation  under  the  streamlined  process  will  not  be  repeated  here.  Figure 

2.3.3-  1  illustrates  how  OWL  considerations  should  enter  the  ASAP;  these  are  identical  to 
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OWL  IN  THE  STREAMLINED  LIFE  CYCLE 


Figure  2.3.3- 1  OWL  Considerations  in  the  ASAP 


those  in  the  traditional  process.  It  is  important  to  emphasize  that  as  the  process  is 
compressed,  so  too  are  the  opportunities  for  and  impacts  of  OWL  consideration.  The  most 
important  phase  of  development  under  the  Streamlined  Acquisition  Process  in  considering 
OWL  inputs  is  Proof  of  Principle.  Incorporation  of  OWL  enhancements  will  be  much 
more  cost  effective  and  efficient  during  and  immediately  after  the  proof  of  principle  phase, 
in  comparison  to  later  in  the  Streamlined  Acquisition  Process.  A  brief  description  of  each 
phase  under  the  streamlined  process,  with  emphases  on  OWL  considerations,  is  presented 
below. 


2.3.3.2  Discussion  of  ASAP  Phases 

The  first  phase  is  the  Requirements  Definition/Technical  Base  Activities.  Activities 
conducted  during  this  phase  are  similar  to  the  program  initiation  activities  under  the 
traditional  process.  They  ultimately  result  in  development  of  a  JMSNS,  or,  as  a  minimum, 
an  O&O  Plan,  and  result  in  approval  to  proceed  into  the  proof  of  principle  phase.  Basic 
program  management  documents  such  as  the  Acquisition  Strategy,  Acquisition  Plan, 
SMMP  and  Test  and  Evaluation  Master  Plan  may  also  be  prepared  during  this  phase. 
OWL  considerations  may  drive  requirements,  as  well  as  the  accomplishment  of  technical 
base  activities,  such  as  research  programs  focused  on  OWL  issues.  OWL  predictions,  and 
the  results  of  OWL  evaluations  of  fielded  systems  which  may  address  similar  mission 
requirements,  will  impact  the  TEMP,  SMMP,  O&O  Plan,  and  AS. 

The  ASAP  calls  for  establishing  a  Technology  Integration  Steering  Committee 
(TISC),  with  the  objective  of  comparing  technological  opportunities  with  emerging 
requirements  (AR  70-1,  Paragraph  7-2c(2)).  OWL  considerations  need  to  be  considered 
by  the  TISC.  TISC  activities  ultimately  develop  solutions  which  are  suitable  for 
consideration  in  hardware  under  proof  of  principle.  Additionally,  "Star"  Reviews  provide 
visibility  and  focus  at  the  general  officer  level  at  the  start  of  proof  of  principle.  OWL 
considerations,  based  on  related  technical  base  activities,  are  issues  for  consideration  by 
the  TISC  and  during  Star  Review. 

The  second  phase  is  Proof  of  Principle.  Proof  of  principle  consolidates  activities 
conducted  during  concept  exploration  and  demonstration/validation  under  the  traditional 
process.  The  phase  permits  the  conduct  of  user  demonstrations  and  experimentation 
employing  brassboard  or  surrogate  systems  in  order  to  prove  out  the  technical  approach 
and  operational  concept.  Proof  of  principle  results  in  a  "go/no-go"  decision  to  proceed  into 


the  development/prove  out  phase.  Documentation  supporting  that  decision  (Milestone  I/II) 
includes  the  CFP,  TEMP,  ILSP,  and  AS.  MANPRINT  and  ILS  issues  are  addressed  on 
the  basis  of  troop  demonstrations  and  experimentation  (AR  70-1,  Paragraph  7-2f(3)). 

Development  of  OWL  requirements,  and  conduct  of  studies  and  analyses  related  to 
establishing  OWL  impacts,  must  receive  strong  emphasis  during  proof  of  principle.  OWL 
evaluations  of  surrogate  systems  and  brassboard  prototypes  serve  as  the  basis  for 
developing  OWL  requirements  as  a  system  enters  the  development/prove  out  phase.  OWL 
predictions,  based  on  systems  currently  in  the  field,  will  also  make  an  important 
contribution  to  understanding  OWL  impacts.  As  with  the  early  phases  of  the  traditional 
MAP,  OWL  considerations  will  have  their  most  cost  effective  impacts  during  proof  of 
principle.  Likewise,  design  features  oriented  to  enhancing  OWL  characteristics  must  be 
identified  during  this  phase.  There  will  be  little  opportunity  for  modifying  designs  in  order 
to  enhance  OWL  characteristics  during  the  development  and  prove  out  phase,  except  under 
the  provision  for  preplanned  product  improvements  (P3I). 

The  third  phase  is  the  Development/Production  Prove  Out  phase.  The 
development/production  prove  out  phase  encompasses  activities  which  are  similar  to  full- 
scale  development  under  the  traditional  process.  System  characteristics  are  demonstrated 
using  hard  tool  prototypes  during  technical  and  operational  testing.  Particular  emphasis  is 
placed  on  ILS,  MANPRINT  (AR  70-1,  Paragraph  7-2g),  and  producibility  engineering  and 
planning.  Documentation,  and  OWL  considerations  to  be  made  during  the  preparation  of 
that  documentation,  is  similar  to  that  required  for  full-scale  development  under  the 
traditional  process.  OWL  predictions  and  the  results  of  OWL  evaluations  are  sources  of 
OWL  data  used  in  preparing  that  documentation.  OWL  evaluation  methodology  is 
employed  during  testing  in  order  to  demonstrate  that  prototypes  meet  required  OWL 
characteristics. 

Throughout  this  phase,  requirements  for  P3I  technology  insertions  are  considered. 
Product  improvements  may  be  made  during  finalization  of  development/production  prove 
out  designs,  or  may  be  delayed  until  further  into  the  production/deployment  phase.  OWL 
enhancements  are  more  likely  to  be  incorporated  in  a  system  which  has  entered  the 
production/deployment  or  the  development/prove  out  phases  as  P3Is.  Based  on  a 
Milestone  III  review  successful  conduct  of  the  development/production  prove  out  phase 
results  in  both  type  classifying  equipment  as  standard,  and  entering  the 
production/deployment  phase. 
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The  final  phase  is  the  Production/Deployment  Phase.  The  production/deployment 
phase  includes  low  rate  initial  production  (LRIP),  first  article  testing  (FAT),  new 
equipment  training,  and  initial  fielding  activities.  Full  rate  production  is  achieved  as  initial 
fielding  is  completed  and  production  models  are  demonstrated  to  achieve  required 
capabilities.  Documentation  required  under  the  streamlined  process  is  similar  to  that 
required  for  the  production/deployment  phase.  Generally,  schedules  for 
production/deployment  under  the  streamlined  process  are  compressed  to  one  and  a  half  to 
two  years.  OWL  evaluation  methodology  may  be  used  to  demonstrate  that  production 
prototypes  meet  required  OWL  characteristics.  OWL  evaluations  of  fielded  hardware  may 
result  in  development  of  requirements  for  product  improvement  as  production/deployment 
is  completed.  These  requirements  normally  would  be  incorporated  as  a  part  of  the  overall 
P3I  program  for  the  system. 

2.3.4  Adoption  of  Non-Developmental  Items  (NDD 

NDI  is  a  candidate  for  fulfilling  any  material  need.  NDI  include  commercially 
available  items,  as  well  as  items  adopted  by  other  services  or  friendly  foreign  nations. 
NDIs  are  considered  in  conjunction  with  market  investigations  accomplished  early  in  the 
development  cycle.  If  pursuing  an  NDI  appears  to  be  an  acquisition  alternative,  a  program 
to  procure,  test,  and  adopt  the  item  is  developed.  There  are  two  general  categories  of  NDI 
(AR  70-1,  Paragraph  7-3d): 

•  Category  A—  Off  the  shelf  items  which  need  no  further  development  or 
modification  in  order  to  achieve  the  required  operational  capability. 

These  items  would  be  expected  to  be  used  in  a  military  environment 
under  the  same  conditions  for  which  they  were  intended  in  commercial 
environment. 

•  Category  B  —  Off  the  shelf  items  requiring  modification  to  hardware 
designs  or  software  in  order  to  operate  in  the  military  environment. 

These  modifications  are  typically  required  to  ruggedize  the  item  or 
enhance  system  survivability. 

Acquisition  Strategies  for  further  development  and  fielding  of  NDI  consider 
modifications  needed  and  the  requirements  to  demonstrate  and  prove  the  suitability  of  the 
equipment  for  military  use.  Adoption  of  an  NDI  does  not  eliminate  the  need  to  examine 
essential  characteristics  to  include  MANPRINT,  systems  safety,  and  logistic  support 
concepts. 
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Conduct  of  NDI  procurement  programs  are  tailored  to  the  requirement  and  the 
availability  of  suitable  commercial  or  foreign  equipment.  NDI  items  which  require 
considerable  ruggedization  with  other  modifications  may  drive  establishing  a  program  with 
features  and  requirements  similar  to  a  hardware  development  program.  Sufficient  testing 
must  be  conducted  to  prove  operational,  maintenance,  and  support  characteristics.  The 
features  of  the  program  are  developed  on  the  basis  of  the  market  analysis  conducted  prior  to 
Milestone  I. 

ILS  and  MANPRINT  issues  vary  from  procvnement  to  procurement.  They  are 
driven  by  the  requirement  and  the  characteristics  of  the  commercial  item  and  the  Army's 
needs.  The  range  of  solutions  varies  from  full  acceptance  of  the  commercial  item,  as  taken 
"from  the  shelf"  with  a  full  commitment  to  future  contractor  maintenance  and  training 
support,  to  incorporation  of  the  items  into  the  Army  training  and  logistic  support  system, 
and  all  combinations  in  between.  All  acceptable  commercially  available  data  is  procured 
and  utilized  for  system  support. 

The  NDI  program  is  tailored,  based  on  the  item  characteristics  and  support 
available,  to  insure  all  requirements  are  met.  For  programs  which  need  hardware 
modifications  and/or  development  of  training  and  support  capabilities,  the  AS  may  be  very 
similar  to  ASAP  of  a  full  system.  The  AS  must  address  MANPRINT  and  ILS  issues  and 
resolutions  must  survive  the  MADP  before  adoption  of  an  NDI. 

2.3.5  Product  Improvements  CAR  70-1.  AR  70-15.  Pam  70-2') 

Product  improvement  is  a  preferred  method  for  responding  to  materiel 
requirements.  They  may  range  from  modifications  to  a  fielded  item  as  a  result  of  a  Product 
Improvement  Program  (PIP)  to  planned  evolutionary  changes  to  a  system  in  development 
(P3I).  A  PIP  is  distinguished  from  a  P3I  in  that  a  PIP  applies  to  systems  which  are  already 
fielded  and  are  no  longer  in  production.  Product  improvements  are  an  appropriate  method 
of  responding  to  workload  deficiencies  which  are  revealed  late  in  the  development  cycle  or 
in  fielded  systems.  PIPs  and  P3Is  are  prioritized  and  integrated  into  the  overall  Army  RDA 
program  in  the  LRRDAP  (see  paragraph  2.3.2.2). 

Product  improvements  may  be  applied  to  systems  for  a  variety  of  reasons.  For  a 
PIP,  these  include:  requirements  to  enhance  human  factors,  safety  or  other  MANPRINT 


related  system  characteristics;  improve  system  technical  characteristics  and  operational 
effectiveness.as  well  as  expand  weapon  system  effective  life.  Preplanned  Product 
Improvements  (P3I)  are  required  to  be  addressed  in  the  requirements  document  for  all 
developmental  systems.  They  may  be  pursued  to:  reduce  development  time;  reduce 
development  risks;  permit  a  timely  response  to  changing  threats;  as  well  as  respond  to 
emerging  technological  opportunities.  P3Is  are  also  appropriate  during  system 
development  to  incorporate  enhancements  to:  system  performance  characteristics  and 
operational  effectiveness;  safety;  human  factors;  and  other  MANPRINT  requirements. 

PIPs  may  be  initiated  by  the  materiel  developer,  the  combat  developer  or  field 
elements.  The  combat  developer  validates  the  need  which  precipitates  the  product 
improvement  requirement.  A  Product  Improvement  Proposal  is  prepared  and  coordinated 
by  the  materiel  developer.  The  Product  Improvement  Proposal  fully  documents  the  need 
for  improvement  and  plans  for  development  and  application  of  the  modifications.  It 
includes  an  acquisition  strategy  and  plans  for  developer  and  user  testing  to  demonstrate  the 
efficacy  of  the  modifications.  Detailed  PIP  procedures  are  established  in  AR  70-15.  The 
PIP  ultimately  produces  a  DA  Modification  Work  Order  (DAMWO),  Depot  Maintenance 
Work  Request  (DMWR)  or  other  approach  to  applying  the  modification  and  documenting  it 
in  the  system  technical  data  package. 

P3I  encourages  an  evolutionary  approach  to  system  design.  That  evolution  is 
described  in  the  acquisition  strategy  for  the  P3I  program  which  are  pursued  in  three  phases: 

•  Phase  I  establishes  how  the  system  needs  to  evolve  throughout  the  life 
cycle  in  order  to  respond  to  future  operational  requirements  and 
technological  opportunities. 

•  Phase  n  incorporates  the  results  of  Phase  I  into  the  basic  system  design. 

•  Phase  III  applies  the  modifications  as  block  (i.e.,  several  system 
changes)  or  individual  changes. 

Systems  which  have  entered  production  and  fielding  must  also  make  provision  for 
the  modification  of  systems  already  in  the  field. 
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2.4  OWL  Issues  in  the  Acquisirion  Process 


2.4. 1  How  OWL  Issues  are  Currently  Addressed 

One  of  the  main  objectives  of  the  document  review  was  to  determine  if  and  how 
operator  workload  issues  (either  physical  or  mental)  were  currently  addressed  as  part  of 
the  Army  MAP.  Not  surprisingly,  there  is  not  much  specific  discussion  or  guidance  given 
in  the  regulations.  There  are  more  citations  when  the  review  is  expanded  to  include 
mention  and  guidance  concerning  areas  related  to  "workload  issues"  (e.g.,  human  factors 
engineering,  manpower,  personnel,  or  other  topic  or  term  that  considers  soldiers  as  well 
as  hardware).  However,  the  connection  and  implications  regarding  OWL  is  more 
tenuous.  This  section  addresses  the  specific  references  to  workload  that  were  found  in  the 
documents  reviewed.  The  manner  in  which  related  topics  are  addressed  in  documents  is 
also  discussed  in  a  more  general  context  of  OWL. 

Documents  from  DoD,  DA  and  subordinate  organizations  were  identified.  Within  each 
document,  other  documents  are  listed  (as  required  publications)  which  are  necessary  for 
complete  understanding  and  implementation.  In  addition,  there  are  also  reference  or  related 
publications  which  may  contain  useful  information  but  are  not  essential  for  complete 
understanding.  By  comparing  the  lists  of  required  and  related  publications,  a  "document 
tree"  may  be  established  as  graphically  displayed  in  Figure  2.4.1-1.  In  this  "tree",  AR  70- 
1  may  be  seen  to  be  the  key  Army  acquisition  document.  It  is  associated  with  other  ARs 
describing  aspects  of  the  acquisition  process  and  with  DoD  high  level  guidance  for  military 
system  acquisition.  AR  70-1  also  requires  more  technically  oriented  ARs  such  as  AR  602- 
1  and  602-2.  With  regard  to  OWL,  the  most  explicit  reference  to  operator  workload  in 
Army- wide  documents  is  contained  in  MIL-H-46855  (cf..  Section  2.4. 1.2).  There  are 
specific  Data  Item  Descriptions  associated  with  this  military  specification  and,  in  turn,  this 
military  specification  is  directly  related  to  the  Human  Factors  Engineering  AR.  This 
indicates  that  these  "workload"-related  DIDs  are  only  referenced  via  the  HFE  AR  602-1. 
The  relationship  shown  in  this  document  tree  could  be  kept  in  mind  during  the  more 
thorough  discussion  of  the  ARs,  DoD  guidance,  and  other  related  documents  which 
follows. 
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REFERENCED/REQUIRED  PUBS 

RELATED  PUBS 


Figure  2.4.1-1.  OWL-  Related  Relationship  Between  Primary  Documents. 
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2.4. 1  ■  1  Army  Regulations 


A  review  of  Army  Regulations  (ARs)  was  undertaken  to  identify  those  that  govern 
system  acquisition  and  any  requirements  for  operator  workload  considerations  as  described 
earlier.  The  key  documents  identified  are  listed  in  Table  2.4. 1-1.  Of  these,  the  first  two 
(AR  70-1,  -10)  are  in  the  Research  and  Development  series.  The  next  two  (AR  71-3,  -9) 
are  in  the  Force  Development  series.  ARs  602-1  and  -2  are  in  the  Soldier-Materiel  Systems 
series.  These  regulations  represent  basic  policy,  guidance,  and  required  formats  in  these 
three  areas. 


Table  2.4. 1-1  Key  Army  Regulations 

AR  Title  Effective  Date 


AR  70-1 

System  Acquisition  Policy  and  Procedures. 

1  Dec  86. 

AR  70-10 

Test  and  Evaluation 

30  Apr  86. 

AR71-3 

User  Testing 

1  Mar  86. 

AR71-9 

Materiel  Objectives  and  Requirements 

20  Mar  87. 

AR  602-1 

Human  Factors  Engineering  Program 

15  Feb  83. 

AR  602-2 

Manpower  and  Personnel  Integration  (MANPRINT) 
in  Materiel  Acquisition  Process 

18  May  87. 

A  brief  summary  of  these  ARs  indicates  the  major  content  and  intention  of  each: 

•  AR  70-1  covers  basic  policies  and  procedures  for  Army  system 
acquisition  and  "emphasizes  front-end  planning  and  tailoring  of  the 
materiel  acquisition  process..."  (p.  3).  The  ASAP  is  introduced  and  its 
policies  and  procedures  described. 

•  AR  70-10  covers  basic  policies  and  procedures  for  test  and  evaluation 
and  provides  information  concerning  test  and  evaluation  for  use  at 
decision  reviews. 

•  AR  71-3  covers  policies  and  assigns  responsibilities  for  user  test  and 
evaluation  and  continuous  comprehensive  evaluation  (C2E)  in  the  MAP. 


•  AR  71-9  covers  policies  and  procedures  and  assigns  responsibilities  for 
preparing  requirements  documents  for  materiel  and  gives  guidance  for 
the  combat  development  process  within  the  MAP. 

•  AR  602- 1  integrates  human  factors  engineering  throughout  the  MAP. 

•  AR  602-2  covers  policies  and  procedures  for  the  MANPRINT  Program 
and  establishes  the  requirement  for  the  System  MANPRINT 
Management  Plan  (SMMP).  MANPRINT  is  an  umbrella  concept  that 
encompasses  HFE,  manpower,  personnel,  training,  health  hazard 
assessment,  and  system  s^ety. 

These  six  documents  do  not  specifically  address  OWL.  Army  Regulation  70-1 
does,  however,  establish  the  policies,  procedures  and  requirements  for  all  applicable  Army 
system  acquisition  programs.  It  calls  out  both  AR  602-1  and  602-2,  and  specifies 
MANPRINT  inputs  into  the  different  phases  of  the  traditional  Life  Cycle  System 
Management  Model  (LCSMM).  Additionally,  it  specifies  that  MANPRINT  considerations 
must  be  included  in  tailored  acquisition  programs  (e.g.,  ASAP  and  NDI)  and  materiel 
improvements  [i.e.,  Engineering  Change  Proposals  (ECP),  Product  Improvement 
Proposals  (PIP),  or  Preplanned  Product  Improvements  (P3I)].  Specifically,  it  provides  that 
"No  acquisition  ...  is  exempt  from  minimal  essential  test  and  evaluation  necessary  to 
verify  the  MANPRINT  .  .  .  characteristics  of  a  system  .  .  .  unless  previous  test  and 
performance  data  or  market  analysis  (information)  is  adequate  for  verifying  operational 
effectiveness  and  suitability  of  die  system"  (p.  27).  Sections  3-8  and  3-9  of  AR  602-2  also 
define  MANPRINT  requirements  for  NDI  and  product  improvements.  In  fact, 
MANPRINT  concerns  alone  can  provide  the  justification  for  a  product  improvement. 
MANPRINT  considerations  are  clearly  related  to  OWL. 

While  not  specific  to  OWL,  higher  level  documentation  calls  out  HFE  requirements 
in  all  phases  of  the  acquisition  process.  AR  602-1  specifies  HFE  requirements  throughout 
the  materiel  life  cycle  and  stipulates  that  the  HFE  program  shall  be  performed  in  accordance 
with  MIL-H-46855B,  thereby  indirectly  establishing  the  requirement  that  OWL  issues 
need  be  addressed.  Also,  program  objectives  such  as  to  ".  . .  Insure,  through  basic  and 
applied  studies  and  research  in  HFE  . . .  that  equipment  designs  and  operational  concepts 
are  compatible  with  the  capabilities  and  limitations  of  operators  and  maintenance  personnel" 
(p.  1-4)  additionally  point  toward  addressing  workload  issues. 

In  concert  with  AR  602-1  is  the  Army's  new  regulation  for  the  implementation  of 
its  MANPRINT  concept,  AR  602-2.  As  it  may  be  recalled,  MANPRINT  is  an  umbrella 
concept  that  encompasses  HFE,  manpower,  personnel,  training,  health  hazard  assessment. 


and  system  safety.  As  such,  it  (AR  602-2)  assumes  the  responsibility  for  coordinating  the 
requirements  of  its  constituent  domains.  Thus,  MANPRINT  policy  provides  that  HFE 
Analysis  will  be  prepared  in  accordance  with  AR  602-1  on  all  Army  major,  designated 
acquisition,  and  in-process  review  (IPR)  programs.  Also,  like  AR  602-1,  AR  602-2 
addresses  the  concept  of  workload  without  specifically  using  the  term  [e.g.,  "... 
Analyses  of  the  work  environment  also  includes  consideration  of  the  physical  and  cognitive 
demands  on  personnel ..."  (p.  3),  and  "...  Ensure  through  studies  and  analyses  and 

basic  and  applied  research  (human  factors  engineering, . . . )  that  equipment  designs  and 
operational  concepts  are  compatible  with  the  limits  of  operators  and  maintainers  defined  in 
the  target  audience  descriptions  ..."  (p.  3-4)].  Thus,  specification  of  MANPRINT 
analysis  requirements  is  equivalent  to  specifying  HFE  and  OWL  analysis  requirements 
within  the  appropriate  problem  domain.  This  is  important  because  higher  level 
documentation,  such  as  AR  70-1  tends  to  address  issues  and  requirements  more  generically 
as  MANPRINT  issues  and  requirements.  This  leaves  the  relevant  lower  level  documents  to 
spell  these  out  the  details  for  the  six  application  areas  of  MANPRINT. 

This  review  altogether  lead  to  the  conclusion  that  attention  to  OWL  concerns  is 
currently  required  for  all  Army  materiel  acquisition  programs.  In  part,  this  conclusion  is 
not  immediately  obvious  because  at  upper  levels  of  acquisition  process  requirements,  the 
OWL  issues  are  subsumed  under  the  more  general  requirements  for  MANPRINT/HFE. 
Also  coupled  with  this  is  the  fact  that  until  recently,  with  the  advent  of  programs  like 
MANPRINT,  HFE  issues  have  not  always  received  their  proper  attention.  This  may  be 
especially  true  for  operator  workload  analysis  which  have  not  been  as  well  developed  as 
other  more  traditional  analyses  of  HFE. 

2.4. 1.2  Military  Specification  MIL-H-46855B  and  Associated  Data  Item 
Descriptions 

The  purpose  of  Military  Specification  MIL-H-46855B  is  to  establish  and  define 
requirements  for  applying  human  engineering  to  the  development  and  acquisition  of 
military  systems,  equipment,  and  facilities.  The  human  engineering  effort  consists  of 
analysis,  design  and  development,  and  test  and  evaluation.  An  outline  of  the  detailed 
requirements  are  given  in  Table  2.4. 1-2. 
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Table  2.4. 1-2  Outline  of  MIL-H-46855B  Requirements 


3.  REQUIREMENTS 

3.1,  General  Requirements 
3.1.1a  Analysis 

3.1.1b  Design  and  Development 
3.1.1c  Test  and  Evaluation 

3.1.2  Human  Engineering  Program  Planning 

3.1.3  Nonduplication 

3.2  Detail  Requirements 

3.2.1  Analysis 

3.2. 1.1  Defining  and  Allocating  System  Functions 

3. 2. 1.1.1  Information  Flow  and  Processing  Analysis 

3.2. 1.1.2  Estimates  of  Potential  Operator/Maintainer 

Processing  Capabilities 

3.2.1. 1.3  Allocation  of  Functions 

3.2. 1.2  Equipment  Selection 

3.2. 1.3  Analysis  of  Tasks 

3.2. 1.3.1  Gross  Analysis  of  Tasks 

3.2. 1.3.2  Analysis  of  Critical  Tasks 

3. 2. 1.3. 3  Workload  Analysis 

3.2. 1.3.4  Concurrence  and  Availability 

3.2. 1.4  Preliminary  System  and  Subsystem  Design 

3.2.2  Human  Engineering  in  Equipment  Detail  Design 

3.2.2. 1  Studies,  Experiments  and  Laboratory  Tests 

3.2.2. 1.1  Mockups  and  Models 

3.2.2. 1.2  Dynamic  Simulation 

3.2.2.2  Equipment  Detail  Design  Drawings 

3.2.2.3  Work  Environment,  Crew  Stations  and  Facilities  Design 

3.2.2.4  Human  Engineering  in  Performance  and  Design 

Specifications 

3.2.2.5  Equipment  Procedure  Development 

3.2.3  Human  Engineering  in  Test  and  Evaluation 

3.2.3. 1  Planning 

3.2.3.2  Implementation 

3.23.3  Failure  Analysis 

3.2.4  Cognizance  and  Coordination 


The  analysis  section  deals  primarily  with  task  analysis,  function  allocation,  and  estimates  of 
potential  operator/maintainer  processing  capabilities.  Specifically,  it  provides: 

"3 .2. 1 .3 .3  Workload  Analysis  -  Individual  and  crew  workload  analysis 
shall  be  performed  and  compared  with  performance  criteria." 
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However,  no  further  information  is  given  as  to  how  to  perform  such  an  analysis 
nor  what  performance  criteria  should  be  used.  This  specific  reference  for  OWL  analysis 
consequently  comes  under  the  domain  of  Human  Factors  Engineering  (HFE).  This 
military  specification  contains  in  its  appendix  an  application  matrix  that  gives  guidelines  as 
to  what  sections  of  the  specification  should  be  applied  during  what  phases  of  the  life  cycle 
as  well  as  what  modifications  should  be  made  depending  on  the  life  cycle  phase.  The  MIL- 
H-46855B  appendix  shows  that  specific  workload  analysis  provision  is  in  effect  in  all 
phases  of  the  life  cycle. 

Data  item  description  (DID)  describes  data  and  prescribes  preparation  instructions 
for  the  data  for  the  analyses  called  out  by  MIL-H-46855.  The  series  of  DIDs  on  human 
factors  engineering  call  for  a  wide  range  of  information  —  the  DIDs  are  listed  in  Table 
2.4. 1-3.  The  specific  DIDs  that  contain  the  requirements  for  implementation  of  this  section 
are  DI-H-7056,  Human  Engineering  Design  Approach  Document  -  Operator  (HEDAD-0), 
and  DI-H-7057,  Human  Engineering  Design  Approach  Document  -  Maintainer.  DI-H- 
7052,  Human  Engineering  Dynamic  Simulation  Plan,  while  not  referencing  Section 
3.2.1.3.3,  does  contain  the  specific  provision  for  the  use  of  dynamic  simulation  for 
workload  analysis.  Of  particular  interest  is  DI-H-7056  because  of  its  specific  application  to 
operators.  Basically,  the  operator-equipment  interfaces  and  the  task  analyses  results  are  to 
be  presented  in  the  HEDAD-0. 

2.4. 1 .3  Aeronautical  Design  Standard-30 

The  Aeronautical  Design  Standard  for  Human  Engineering  Requirements  for 
Measurement  of  Operator  Workload  (ADS-30)  was  the  only  Army  document  found  that 
dealt  specifically  with  OWL.  The  proponent  is  the  U.S.  Army  Aviation  Systems 
Command,  St.  Louis,  MO.  ADS-30  "establishes  the  requirement  for  a  comprehensive, 
broadly-based  workload  assessment"  to  identify  workload  "chokepoints"  in  materiel 
systems  (i.e..  Army  aviation  systems).  This  standard  provides  for  the  workload 
assessment  to  be  carried  out  throughout  the  design  and  development  portions  of  the 
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Table  2.4. 1-3  Human  Factors  Engineering  Data  Item  Descriptions 


Number 

Dr-H-7051 

DI-H-7052 

DI-H-7053 

DI-H-7054 

DI-H-7055 

DI-H-7056 

DI-H-7057 

DI-H-7058 

DI-H-7059 


Title 

Human  Engineering  Program  Plan 
Human  Engineering  Dynamic  Simulation  Plan 
Human  Engineering  Test  Plan 
Human  Engineering  System  Analysis  Report 
Critical  Task  Analysis  Report 

Human  Engineering  Design  Approach  Document— Operator 

Human  Engineering  Design  Approach  Document— 
Maintainer 

Human  Engineering  Test  Report 
Human  Engineering  Progress  Report 


acquisition  process.  Types  of  OWL  techniques  are  discussed  and  management  methods  for 
assessment  by  contractors  are  discussed. 

2.4. 1 .4  Department  of  Defense  Documents 

Three  Department  of  Defense  (DoD)  level  documents  pertaining  to  system 
acquisition  were  reviewed  for  guidance  regarding  OWL.  (These  are  listed  in  Table  2.4.1- 
4.)  DoD  Directive  (DoDD)  5000.1,  the  first  of  these,  presents  DoD  acquisition  policy  for 
major  systems  or  major  modifications  to  existing  systems.  Broad  guidance  for  technical 
issues  is  included  in  the  list  of  acquisition  management  principles  and  objectives.  The  most 
applicable  principle  to  OWL  issues  is  that  improved  readiness  and  sustainability  are  primary 
objectives,  with  operational  suitability  of  equal  importance  to  operational  effectiveness. 
Operational  effectiveness  is  the  overall  degree  of  mission  accomplishment  of  the  system. 
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Operational  suitability  is  the  degree  to  which  the  system  can  be  placed  satisfactorily  in  field 
use,  with  consideration  given  to  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors, 
manpower  supportability,  logistic  supportability  and  training  requirements.  Operational 
suitability  includes  the  ability  of  the  soldier  to  operate,  maintain  and  support  the  system, 
which  would  include  OWL. 


Table  2.4. 1-4  Department  of  Defense  Documents 


Document 

Subject 

Date 

DoD  Directive 

5000.1 

Major  Systems  Acquisitions. 

% 

12  Mar  86. 

DoD  Instruction 

5000.2 

Major  System  Acquisition  Procedures 

12  Mar  86. 

DoD  Directive 

5000.3 

Test  and  Evaluation 

12  Mar  86. 

DoD  Instruction  (DoDI)  5000.2  describes  the  procedures  to  implement  DoDD 
5000.1.  Workload  issues  are  never  specifically  addressed,  but  might  become  topic  areas 
included  in  a  program  review  or  in  the  program  documents  used  in  support  of  DoD  level 
decisions.  DoDD  5000.3  provides  policy  and  guidance  for  test  and  evaluation  in  DoD  and 
provides  guidance  for  the  TEMP.  Again,  specific  issues  are  not  addressed  in  this  directive, 
although  "use  of  properly  validated  analysis,  modeling  and  simulation  is  strongly 
encouraged,  especially  during  early  development  phases..."  (p.  3).  An  important  aspect  of 
testing  and  evaluation  is  addressing  critical  issues  that  have  been  identified  or  may  arise 
throughout  the  MAP.  The  DoD  Directives  (5000.1-5000.3)  do  not  address  specific  OWL 
but  provide  high  level  policy  that  operational  suitability  is  important  and  test  and  evaluation 
should  address  important  issues. 
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2.4. 1 .5  Other  Sources 


Table  2.4. 1-5  lists  several  further  Army  sources  which  that  address  OWL.  The 
MIL-STD  1472C,  the  first  of  these,  is  the  basic  military  standard  for  human  engineering 
design  criteria.  The  general  requirements  for  equipment  design  include  the  principle 
"Design  shall  be  such  that  operator  workload,  accuracy,  time  constraint,  mental  processing 
and  communication  requirements  do  not  exceed  operator  capabilities"  (p.  13).  Similarly, 
software  is  to  be  designed  to  minimize  task  complexity  and  memorization.  Operator 
response  times  will  be  with  operational  task  limits  (p.  242).  The  term  "workload"  appears 
in  one  other  place;  on  p.  63,  it  is  stated  that  the  "distribution  of  work  load"  should  be  such 
that  none  of  the  operator  limbs  are  overburdened.  This  is  workload  in  the  functional 
allocation  sense.  Interestingly,  "workload"  does  not  appear  in  the  index.  MIL-STD- 
1472C  is  a  basic  document  where  designers  might  look  for  information,  but  does  not 
provide  any  information  or  suggestions  specifically  addressing  OWL  (with  the  few 
exceptions  noted  above). 

Two  sources  were  provided  to  us  by  individuals  at  TECOM.  The  TECOM 
Pamphlet  602-1  (Vol.  1),  in  particular,  describes  how  to  design  subjective  opinion  tests 
and  was  identified  by  those  individuals  as  "what  was  used  to  assess  workload  in  technical 
tests."  The  second  source  was  the  Test  Operating  Procedure  (TOP)  1-2-610  which 
provides  detailed  design  criteria  against  which  to  test  equipment.  One  of  the  procedures 
included  is  a  Workload  Assessment  (p.  131)  which  suggests  a  time-line  analysis  and 
supplementing  these  observations  with  subjective  questions.  A  Workload  Assessment  form 
is  included  with  headings  such  as  critical  task,  time  required,  additional  tasks  conducted 
simultaneously,  effects  of  time  delays  in  task  completion,  overload  problems  and  underload 
problems.  This  is  addressing  "workload"  in  the  context  of  sharing  or  consolidating  the 
tasks  to  be  accomplished. 
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Table  2.4. 1-5  Other  Army  OWL  Sources 


Document  Tide_ Date 

MIL-STD-1472C 

Human  Engineering  Design  Criteria  2  May  8 1 

for  Military  Systems,  Equipment  and 
Facilities 


ADS-30  Human  Engineering  Requirements  17  Nov  86 

for  Measurements  of  Operator 
Workload 

TECOM  Pam  602-1 

Questionnaire  and  Interview  Design  25  Jul  75 
(Vol  I)  (Subjective  Testing  Techniques) 

TOP  1-2-610  Human  Factors  Engineering  Data  20  Nov  83 

Guide  for  Evaluation  (HEDGE) 


Several  documents  originally  identified  as  relevant  were  unavailable  for  the  present 
review  because  they  are  under  revision  or  out  of  stock  at  the  document  distribution  center. 
The  documents  include: 

•  MIL-HDBK-759  "Human  Factors  Engineering  Design  for  Army 
Materiel" 

•  AR  10-41  "Organization  and  Functions,  U.S.  Army  Training  and 
Doctrine  Command" 

•  AR  15-14  "Systems  Acquisition  Review  Council  Procedures 

•  MIL-STD- 1388-1/2  "Logistic  Support  Analysis  "/"Logistic  Support 
Analysis  Record" 

Efforts  to  obtain  and  review  these  particular  documents  as  well  as  identify,  obtain 
and  review  other  relevant  documents  to  enhance  our  understanding  of  OWL  requirements 
will  continue.  It  is ,  however,  believed  that  the  present  review  has  essentially  revealed  the 
status  of  OWL  in  the  Army  MAP. 
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2.4.2  Terminology 


The  term  "workload"  carries  a  multitude  of  meanings  within  the  military 
community.  For  example,  HARDMAN-type  manpower,  personnel  and  training  (MPT) 
analyses  specify  workload  and  workload  analysis  for  system  operators  and  maintainers. 
Within  this  context,  however,  workload  is  defined  as  the  number,  frequency  and  durations 
of  activity-based  tasks,  performed  by  a  specific  number  of  personnel  of  particular  MOS's, 
skill  levels  and  paygrades.  The  output  of  workload  analyses  are  numbers  and  grade 
qualifications  of  personnel  necessary  to  operate  and/or  maintain  a  system.  This 
performance/manhour  based  interpretation  is  much  more  restricted  than  that  which  is  taken 
here  or  that  is  necessary  to  fully  address  OWL  issues.  Within  other  contexts,  it  is  clear  that 
workload  does  not  refer  to  cognitive/physical  underload  or  overload,  but  rather  to  -  task- 
based  manning  considerations.  It  seems,  then,  that  care  must  be  taken  to  clearly  discuss 
exactly  what  is  being  discussed  when  using  terms  like  "workload"  and  "workload 
analysis".  Certainly,  manpower  considerations  are  closely  tied  to  potential  "cognitive 
overload"  (see  section  2.4.3),  but  they  are  different  and  should  be  clearly  differentiated. 
Care  must  also  be  taken,  as  intended  here,  when  addressing  a  military  audience  to  insure 
that  the  proper  framework  for  discussing  perceptual,  cognitive,  psychomotor  workload,  is 
established  up  front. 

2.4.3  MANPRTNT 


The  Army  Manpower  and  Personnel  Integration  (MANPRINT)  initiative  focuses 
on  the  soldier-in-the-loop  and  front-end  analysis  in  the  acquisition  process.  As  noted 
earlier,  MANPRINT  seeks  to  integrate  six  areas  that  are  concerned  with  the  soldier  into 
the  materiel  acquisition  process  so  that  soldier  needs  and  abilities  are  considered  early  in  the 
process.  The  six  areas,  called  domains,  are  Manpower,  Personnel,  Training,  Human 
Engineering,  Health  Hazards  Assessment,  and  System  Safety. 

2,4.3. 1  Interrelationships  between  OWL  and  the  Domains 

Manpower,  personnel  and  training  (MPT)  are  critical  areas  in  the  Army  MAP. 
Manpower  is  concerned  with  force  structure  and  deals  with  how  many  people  and  of  what 
Military  Occupational  Speciality  (MOS)  are  needed  to  operate,  maintain,  and  support 
materiel.  These  are  sometimes  referred  to  as  the  "spaces."  Personnel  issues  deal  with  the 


2-35 


kind  of  people  needed  to  operate,  maintain,  and  support  materiel.  People  in  this  context 
are  recognized  as  possessing  different  levels  of  intelligence  and  skill,  as  well  as  different 
personality  attributes.  Personnel  issues  are  sometimes  referred  to  as  the  "faces",  implying 
the  individual  characteristics  of  the  soldier.  Training,  of  course,  is  the  instruction  of  the 
soldier  in  specific  skills  and  procedures  needed  to  perform  necessary  tasks.  Training  is 
done  in  schools  and  in  units  and  many  methodologies  are  used.  Each  of  these  areas  must  be 
addressed  in  the  development  of  Army  requirements  and  are  not  synonymous  with  OWL. 
However,  there  are  interrelationships  between  OWL  and  MPT  that  should  be  kept  in  mind 
as  this  effort  proceeds. 

One  relationship  between  manpower  and  OWL  is  the  term  "workload"  (as 
discussed  in  Section  2.4.2).  For  many  tasks,  it  is  not  inappropriate  to  conclude  that  if 
there  is  too  much  for  one  person  to  do  in  a  certain  amount  of  time  to  specific  criteria  levels, 
then  having  two  people  do  the  job  will  take  care  of  the  problem.  However,  this  relatively 
simple  addition  of  more  people  (assuming  that  the  additional  people  with  the  needed 
abilities  are  available)  will  not  solve  every  problem.  The  process  of  perceiving  and 
processing  must  ultimately  rest  on  single  operators  in  many  circumstances.  Adding  another 
person  in  this  case  would  not  help. 

The  distinction  between  between  manpower  and  OWL  concerns  may  be  made  by 
questioning  (1)  whether  the  task(s)  that  are  creating  the  "workload"  are  of  the  kind  that  can 
solved  by  just  adding  another  person  or  (2)  is  the  nature  of  the  task  such  that  it  must  be 
done  by  an  individual  and  it  requires  too  much  in  too  short  a  time  period. 

Personnel  issues  are  concerned  with  the  individual  characteristics  of  the  soldiers. 
The  interrelationship  between  OWL  and  personnel  issues  involve  such  areas  as  the  trade 
offs  between  intelligence  and  "quality"  as  identified  by  the  ASVAB  (i.e.,  mental  categories 
I-IV)  and  the  degree  of  soldier  perceptual,  cognitive,  or  psychomotor  loading.  Trade  offs 
may  need  to  be  identified  depending  on  the  types  of  soldiers  available  because  of  this 
interaction  of  personnel  characteristics  and  system  design. 

Training  is  another  area  with  which  OWL  is  interrelated.  Increased  training  gives 
the  soldier  knowledge,  skills,  and  practice  in  the  required  tasks.  Additional  training  may  be 
and  is  frequently  treated  as  lh£  solution  to  overcome  inadequate  performance.  However, 
training  often  may  not  be  effective  in  reducing  workload,  or  a  cost  effective  way  of  and 
enhancing  performance  (Hart,  1986).  In  order  to  adequately  control  the  more  cognitive 
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workload,  there  may  need  to  be  trade  offs  made  between  training  and  the  quality  of  soldier 
chosen  as  the  operator.  Another  alternative  is  to  affect  the  hardware  design  as  part  of  HFE. 

Human  Factors  Engineering  (HFE)  is  concerned  with  the  engineering  design  of 
equipment  and  the  soldier-machine  interface  so  that  system  performance  (including  the 
human  element)  is  maximized.  OWL  issues  are  interrelated  with  HFE  in  the  design  of  the 
equipment,  specifically  the  soldier-machine  interface  design.  If  an  interface  has  been 
designed  well,  the  ease  with  which  the  operator  can  perceive  information  or  perform 
motor  tasks  may  be  optimal,  thereby  reducing  workload.  An  unthoughtful  and  poorly 
designed  interface  may  be  the  major  factor  in  creating  a  workload  intensive  task.  Human 
factors  engineering  solutions  should  certainly  be  among  the  first  pursued  when  overload 
problems  are  identified. 

Health  hazard  assessment  (HHA)  is  concerned  with  any  condition  inherent  to  the 
use  of  equipment  that  may  cause  degradation  of  job  performance,  chronic  disability  or 
death.  Health  hazards  include  toxic  substances,  vibration,  noise,  temperature  extremes  and 
psychological  stressors.  Although  this  last  area  is  not  universally  considered  a  health 
hazard,  psychological  stressors,  such  as  confined  spaces,  isolation,  sleep  deprivation,  and 
sensory/cognitive  overload,  may  cause  serious  degradation  or  chronic  disability  in  job 
performance.  There  is  currently  no  overall  Army  program  addressing  these  psychological 
stressors  from  a  HHA  perspective.  However,  health  hazard  assessors  try  to  identify 
potential  problems  early  in  the  acquisition  process  and  raise  a  flag  that  this  issue  must  be 
considered  as  the  MAP  continues  (LTC  B.  Leibrecht,  USAARL,  personal  communication, 
30  April  1987).  OWL  should  be  an  issue  from  the  HHA  perspective  in  the  acquisition 
process. 

System  safety  (SS)  is  concerned  with  identifying  and  eliminating  or  reducing  the 
risks  associated  with  system  (particularly  hardware)  characteristics  that  may  cause  injury  or 
death.  The  results  of  hardware  failure  (e.g.,  electrical  shorts  or  restraint  harnesses 
breaking)  are  of  particular  concern.  OWL  issues  are  related  to  the  safety  of  the  system  only 
and  to  the  extent  these  risks  intrude  and  occupy  the  operator. 

It  can  be  concluded  from  the  above  discussions  that  operator  workload  is  related  to 
all  six  areas  of  MANPRINT.  It  is  not  clearly  synonymous  with,  nor  falls  under  the 
specific  purview  of,  any  particular  domain.  However,  the  interrelationships  between 
OWL  and  the  MANPRINT  domains  are  important  considerations  in  developing  system 
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requirements  and  design.  Considerations  of  these  interrelationships  will  identify  MPT, 
HFE,  HHA  or  SS  trade-offs  that  may  be  made  in  an  effort  to  control  OWL  as  system 
requirements  and  design  are  defined. 

As  specified  by  the  Army  in  its  training  courses  on  MANPRINT,  there  are  several 
tools  to  be  employed  by  the  six  MANPRINT  domains.  Among  these  is  workload  analysis, 
which  the  Army  indicates  is  to  be  used  during  all  phases  of  the  acquisition  process  to 
answer  such  questions  as: 

•  Which  design  alternative  is  the  best? 

•  What  training  will  be  required? 

•  Can  operators  perform  all  functions  effectively? 

•  What  design  inadequacies  exist  that  must  be  rectified? 

These  questions  as  well  as  others  are  raised  during  the  various  phases  of  the 
acquisition  process  as  appropriate,  and  workload  estimation  and  measurement  techniques 
must  be  developed  that  can  provide  timely  answers. 

2.4.3.2  System  MANPRINT  Management  Plan 

The  System  MANPRINT  Management  Plan  (SMMP)  is  the  "planning  and 
management  guide  and  an  audit  trail  to  identify  tasks,  analyses,  trade-offs,  and  decisions 
that  must  be  made  to  address  MANPRINT  issues  during  the  materiel  development  and 
acquisition  process"  (AR  602-2,  p.  12). 

The  SMMP  is  initiated  very  early  in  the  acquisition  process  by  the  combat  or 
training  developer  and  requires  consideration  of  concerns  and  questions  that  may  affect 
soldier  performance  in  Army  equipment.  This  is  an  appropriate  and  logical  place  to  include 
OWL  concerns  and  has  the  important  attribute  of  being  initiated  at  the  very  outset  of 
materiel  requirements  development.  The  SMMP  is  to  be  started  prior  to  the  program 
initiation,  and  most  likely  will  be  developed  concurrently  with  the  O  &  O  Plan.  Even  at  this 
point,  OWL  concerns  can  be  raised  and  methods  to  answer  questions  and  address 
concerns  can  be  suggested. 

The  format  for  the  SMMP  is  given  in  Appendix  B  of  AR  602-2.  The  five  major 
sections  include: 

•  a  summary  of  MANPRINT  strategy 
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•  a  description  of  the  proposed  system,  with  the  acquisition  strategy, 
agencies  involved  and  any  existing  guidance  also  described 

•  a  description  of  the  MANPRINT  strategy  including  the  objectives,  data 
source  availability,  and  planned  MANPRINT  analyses 

•  any  issues  or  areas  of  concern  which  have  been  identified 

•  the  tabs,  which  include  list  of  data  sources,  MANPRINT  milestone 
schedule,  task  description,  questions  to  be  resolved,  and  list  of  all 
organizations  with  which  the  SMMP  must  be  coordinated. 

Although  the  regulation  requiring  the  SMMP  is  new  (17  April  1987),  some 
guidance  is  available  through  the  SMMP  Procedural  Guide  (1986).  It  is  expected  that  as 
more  experience  is  gained  by  those  who  prepare  the  SMMP,  the  SMMPs  will 
increasingly  address  issues  in  the  MANPRINT  areas  and  provide  a  useful  management 
plan  to  control  factors  such  as  OWL. 

The  SMMP  has  several  sections  which  provide  opportunities  to  address  OWL 
concerns  early  and  throughout  the  MAP.  In  particular,  the  Concerns  section  (paragraph  4) 
is  the  place  to  discuss  any  identified  issues  in  the  system  development.  These  concerns  are 
those  that  should  be  monitored  throughout  the  MAP.  Further,  the  Questions  to  be 
Resolved  (Tab  D)  are  the  detailed  questions  that  need  to  be  answered  to  address  the 
concerns  identified  in  Paragraph  4.  These  questions  should  be  detailed  and  specific.  In 
some  ways,  development  of  the  Tab  D  questions  will  lead  to  the  analyses  that  need  to  be 
done  in  order  to  obtain  sufficient  information  to  answer  the  questions  (these  analyses  are  to 
be  presented  as  part  of  Paragraph  3b  as  well  as  the  identification  of  predecessor  or 
reference  systems  and  what  kind  of  data  is  expected  to  be  available  for  use).  Tab  A  is  also 
identified  as  the  place  to  list  all  potential  data  sources  in  all  the  MANPRINT  domains  and 
should  also  include  those  relevant  to  OWL.  A  description  of  the  tasks  to  be  done  in  support 
of  MANPRINT  efforts  are  to  be  presented  in  Tab  C.  These  descriptions  include  the 
rationale,  resources  needed,  time  to  complete,  and  responsible  agencies. 

Clearly,  if  an  awareness  of  and  sensitivity  to  OWL  issues  can  be  developed  by 
those  preparing  the  SMMPs,  then  their  format  should  provide  the  means  to  surface  broad 
concerns  about  workload  issues  and  well  as  the  specific  questions  that  need  to  be 
investigated  in  order  to  adequately  address  the  stated  concerns.  The  identification  of 
predecessor  system  and  data  list  will  directly  affect  the  types  of  OWL  predictive  and/or 
evaluative  assessments  that  can  be  conducted.  Similarly,  it  will  affect  the  timeliness  with 
which  such  assessment  can  be  performed  in  the  sense  that  well-documented  data  sources 
and  known  availability  will  faciUtate  its  gathering  and  application. 
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Any  key  OWL  issues  or  concerns  should  also  be  included  in  the  summary 
(Paragraph  1)  which  presents  an  overview  of  MANPRINT.  The  summary  will  be  the 
portion  most  often  read  by  decision  makers  and  will  give  visibility  to  the  key  issues.  The 
status  of  key  issues  can  be  monitored  and  managed  as  the  SMMP  is  continually  updated 
with  current  information. 

Another  aspect  of  the  SMMP  that  is  of  interest  in  the  control  of  OWL  is  the  role  of 
the  MANPRINT  Joint  Working  Group  (MJWG).  Although  the  SMMP  is  the 
responsibility  of  the  Combat  Developer  (i.e.,  TRADOC),  it  is  to  be  developed  in 
conjunction  with  the  Materiel  Developer  (i.e.,  AMC).  The  MANPRINT  Joint  Working 
Group  (MJWG)  is  the  group  of  people  that  work  together  to  create  the  SMMP  and  includes 
representatives  from  the  Combat  Developer,  the  Materiel  Developer,  and  other 
organizations  that  are  involved.  The  SMMP  should  consequently  have  inputs  from  all 
interested  organizations  and  they  can  play  a  part  in  the  lists  of  concerns,  questions,  and 
tasks  to  be  accomplished. 

2.4.4  Identified  Army  Projects 

During  the  document  review,  other  procedures  and  analyses  that  have  been 
developed  for  the  Army  and  that  may  be  useful  in  this  effort  were  identified.  As  we 
continue  our  investigations,  existing  procedures  or  information  that  can  be  used  in 
workload  analyses  will  be  utilized  to  the  fullest  extent  possible.  Some  of  the  identified 
sources  and  the  potential  use  in  OWL  assessment  are  delineated  in  the  following. 

2.4.4. 1  Early  Comparability  Analysis  fECAV 

The  Early  Comparability  Analysis  (ECA)  Procedural  Guide  (1986)  summarized 
that  "the  ECA  methodology  is  based  on  a  'lessons  learned'  approach  to  the  design  of  a 
conceptual  system"  (p.  1).  For  ECA,  a  predecessor  (or  reference)  system  is  identified 
with  which  to  compare  the  conceptual  system.  Relevant  MOSs  are  identified,  task  lists  for 
those  MOSs  are  obtained  (or  derived  if  not  available).  Subject  Matter  Experts  (SMEs)  are 
also  consulted  to  assign  "difficulty"  ratings  to  tasks  using  six  task  criteria;  percent 
performing,  task  learning  difficulty,  task  performance  difficulty,  frequency  rate,  decay  rate, 
and  time  to  train.  Those  tasks  costly  in  MPT  resources  are  identified  and  solutions  are 
proposed  to  eliminate  or  reduce  the  cost  of  these  "high  drivers." 
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EGA  uses  a  combination  of  several  analytic  techniques  to  identify  MPT  resource¬ 
intensive  tasks  (comparison,  SMEs,  task  analysis) .  As  discussed  previously,  MPT  issues 
are  interrelated  with  OWL  issues  and  their  treatment  in  the  context  of  an  analysis  already  in 
place  can  yield  OWL  information  very  early  in  the  MAP.  This  could  enhance  the 
development  of  OWL  analyses  as  EGA  is  designed  for  use  by  combat  developers  as  an  in- 
house  tool,  therefore  it  is  not  intended  to  be  a  highly  sophisticated  tool  for  use  only  by 
technical  experts.  OWL  estimations  have  been  previously  made  based  upon  comparability 
(e.g.,  Shaffer  et  al..  1986),  SME  (Zachary,  1980),  and  task  analysis  (e.g..  Stone  et  al.. 
1985). 


2.4.4.2  Hardware  vs.  Manpower  (HARDMANl 

The  HARDMAN  Gomparability  Analysis  is  a  "structured  approach  to  the 
determination  of  the  Manpower,  Personnel  and  Training  (MPT)  requirements  of  a  weapon 
system  in  the  earliest  phases  of  its  development"  (Mannle,  Guptill,  and  Risser,  1985). 
HARDMAN  is  primarily  an  MPT  analysis  and  sophisticated  comparison  methodology  is 
used  to  derive  estimates  for  MPT.  It  does  produce  task  analyses  with  time-lines  that  could 
be  used  for  certain  OWL  estimation  methodologies  (e.g..  Stone  et  al..  1985).  However, 
HARDMAN  is  a  sophisticated  tool  -  currently  only  one  company  performs  the  analyses 
for  the  Army  —  and  it  is  expensive  to  do.  Therefore,  HARDMAN  will,  most  likely,  not  be 
available  for  use  on  all  systems  and  may  not  be  expected  to  provide  a  broad  basis  for 
predictive  analyses  of  OWL. 

2.4.4.3  Logistic  Support  Analysis  (LSAl 

The  Logistic  Support  Analysis  (LSA),  as  described  by  AR  700-127,  is  performed 
to  identify  existing  or  proposed  support  structure  and  requirements,  as  well  as  apply 
Integrated  Logistics  Support  (ILS)  and  MANPRINT  influence  in  system  design  and 
selection.  LSA  is  required  in  all  acquisition  programs.  As  part  of  the  LSA,  certain  tasks  are 
required  of  both  the  combat  and  materiel  developers.  Such  tasks  as  use  studies  (Task  201), 
comparative  analyses  (Task  203),  and  task  analyses  (Task  401)  are  required  in  accordance 
with  MIL-STD-1388.  The  LSAR  may  be  a  useful  source  of  data  for  maintainer  workload 
estimation  and  evaluation.  Further  investigation  is  needed  to  determine  what  data  would 
actually  be  available  for  use  for  assessments  of  OWL. 
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2.4.4.4  Human  Resources  and  Test  Evaluation  System  (HRTES') 


The  Human  Resources  and  Test  Evaluation  System  (HRTES)  (Kaplan,  Crooks, 
Sanders  and  Dechter,  1984)  is  a  set  of  procedures  designed  to  assist  a  test  planner  to 
evaluate  operator  and  maintainer  performance  in  an  operational  test  of  an  Army  system.  The 
HRTES  procedure  takes  the  test  planner  through  a  series  of  steps  from  general  system 
issues  to  the  highly  specific  human  performance  issues.  For  example,  a  set  of 
considerations  would  be  to  1)  define  mission  performance  2)  define  performance 
conditions  such  as  weather,  3)  identify  system  performance  issues  and  criteria  and  then  4) 
identify  human  tasks  and  performance  criteria  for  each  task.  Interspersed  through  this 
analysis  is  the  reminder  of  the  need  for  planning  and  for  identifying  the  techniques  for 
measuring  each  of  the  criteria.  The  HRTES  procedures  have  potential  utility  for  OWL 
assessment  with  regard  to  identifying  and  defining  system  characteristics  for  use  in  the 
selection  of  individual  techniques. 

2.4.4.5  MANPRINT  Data  Base 

A  MANPRINT  data  base  is  currently  under  development  at  the  U.S.  Army  Materiel 
Readiness  Support  Activity.  The  main  purpose  of  the  data  base  is  to  organize  MPT,  HFE, 
HHA  and  SS  data  for  use  in  comparative  analyses.  The  data  base  will  contain  historical 
data  organized  by  end  item.  The  MPT  portion  will  contain  such  items  as  RAM  data,  MOSs 
related  to  the  end  item,  manhours  and  tasks.  The  HFE,  HHA,  and  SS  area  will  be 
addressed  in  more  of  a  "lessons  learned"  approach,  with  any  problems  being  identified  and 
solutions  (if  any)  given.  The  plan  is  to  have  the  programming  of  the  data  base  on  line  by 
4th  Quarter,  FY  88.  However,  it  may  be  2  or  3  years  before  the  loading  of  the  data  is 
complete  and  the  data  base  is  accessible  to  outside  users. 

A  key  problem  in  the  data  base  development  is  the  availability  of  operator  data.  So 
far  only  the  maintainer  is  fully  documented,  primarily  because  generic  functions/subtasks 
are  more  easily  defined  for  maintainers.  It  is  much  more  difficult  to  define  generic 
functions/subtasks  for  operators  (Mr.  G.  Tarber,  USAMRSA,  personnel  communication, 
8  May  1987). 

As  the  data  base  is  further  developed  and  becomes  accessible,  it  may  provide  a 
good  source  of  data  for  workload  and  comparability  analyses.  Currently,  however, 
operators  are  not  addressed  and  there  is  not  a  firm  plan  on  how  to  do  so. 
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2.4.4.6  Summary 


These  procedures  have  been  identified  as  potential  sources  of  information  or  data 
for  use  in  OWL  assessments.  The  descriptions  of  the  procedures  and  how  they  may  be 
used  for  OWL  assessment  are  only  preliminary  judgments.  Additional  methods  and 
procedures  as  well  as  application  potentials  may  be  expected  to  evolve  because  of  the 
increasing  interest  in  programmatic  effort  (e.g.,  MANPRINT). 

2.4.5  Conclusions  and  Recommendations 


A  number  of  conclusions  can  be  drawn  from  the  document  review.  Some  of  these 
conclusions  will  have  an  impact  on  the  future  tasks  of  the  OWL  project  in  directing  how  the 
guidance  for  OWL  assessment  are  written  (i.e.,  the  handbooks).  The  major  conclusions 
and  recommendations  are: 

•  In  general,  there  is  a  void  in  Army  documents  on  the  topic  of  OWL. 

•  OWL  assessment  is  required  (via  MIL-H-46855),  but  indirectly  through 
the  areas  of  HFE  and  MANPRINT. 

•  There  do  not  seem  to  be  any  inconsistencies  regarding  OWL,  primarily 
because  there  isn't  much  available. 

•  The  intent  to  consider  the  soldier  in  materiel  acquisition  is  clear  in  high 
level  DoD  Directives. 

•  Clear  distinctions  between  types  of  workload  must  be  made. 

•  Effort  should  be  made  to  make  use  of  existing  projects  as  much  as  is 
appropriate. 

•  ASAP,  NDI  and  product  improvement  strategies  will  make  early 
attention  to  OWL  even  more  critical. 

•  MANPRINT  provides  a  framework  on  which  to  build  and  provides 
places  to  address  OWL  issues  (e.g.  the  ROC  and  the  SMMP). 

•  The  MANPRINT  portion  of  the  ROC  is  an  appropriate  place  to  address 
soldier-in-the-loop  requirements  (e.g.,  OWL)  for  the  new  equipment. 

•  The  SMMP  would  be  a  useful  vehicle  to  focus  attention  on  potential 
OWL  problems  and  devise  plans  to  address  OWL  throughout  the  MAP. 

The  review  of  the  documents  provided  a  useful  means  to  understand  the  current 
Army  requirements  and  how  issues  concerning  operator  workload  are  addressed.  The  lack 
of  specific  guidance  was  not  really  surprising  -  there  has  been  greater  awareness  in  recent 
times  because  of  the  technological  advances  being  included  in  Army  materiel.  The  identified 
lack  emphasizes  the  timeliness  and  importance  of  the  current  OWL  project  effort. 
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3.  ASSESS  USER  NEEDS 


3.1  Introducrion 


In  the  effort  to  fully  understand  the  Army  MAP  and  how  OWL  issues  are 
addressed,  a  review  of  written  documents  was  conducted  as  described  in  Section  2 
subsequent  to  beginning  the  interviews.  A  better  understanding  of  the  process  and  issues 
was  obtained  by  talking  to  the  people  who  are  actually  involved  in  the  process.  The 
purposes  were  to  further  our  understanding  of  the  Army  MAP  and  learn  how  OWL  is 
currently  considered.  These  discussions  provided  the  opportunity  to  identify  the  concerns 
of  the  Army  community  with  respect  to:  workload  (and  related  items  they  cared  to  share); 
what  guidance  or  tools  would  be  most  helpful  to  them;  and  how  things  actually  worked  as 
opposed  to  the  written  descriptions  that  had  been  reviewed. 

This  section  will  describe  our  approach  to  obtaining  information  about  Army  user 
needs  and  will  report  the  information  obtained.  With  the  exception  of  the  information 
received  via  the  questionnaires  (see  Section  3.4),  the  information  presented  has  been 
obtained  from  discussions  and  has  been  integrated  to  reflect  general  concerns  and 
suggestions,  rather  than  identifying  specific  users  and  their  comments.  This  section 
overviews:  our  approach;  user  concerns  as  expressed  in  discussions  and  questionnaires; 
user  suggestions  concerning  the  consideration  of  OWL  during  the  MAP;  as  well  as  the 
handbooks  to  be  produced  during  this  contract  effort.  Finally,  plans  for  follow-up 
assessment  of  user  needs  will  be  discussed. 

3.2  Approach 


3.2. 1  Army  Organizations  Visited 

The  Army  organizations  with  whom  we  spoke  and  the  date(s)  visited  are  presented 
in  Table  3.2.1-1.  In  addition,  the  DoD  HFE  Test  and  Evaluation  Technical  Advisory  Group 
was  briefed  on  7-8  Apr  87  at  NASA-Langley,  VA.  Researchers  at  NASA-Langley  also 
briefed  us  regarding  their  recent  and  continuing  work  in  the  area  of  mental-state  estimation. 
Their  work  relates  directly  to  workload  assessment  and  will  be  included  in  our  Task  3 
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Table  3.2. 1-1  Army  Organizations  Visited 


Date 

Organization 

1  Apr  87 

U.S.  Army  Materiel  Command 

Alexandria,  VA 

1  Apr  87 

U.S.  Army  Soldier  Support  Center  —  National 

Capitol  Region 

Alexandria,  VA 

2  Apr  87 

U.S.  Army  Operational  Test  and  Evaluation  Agency 

Falls  Church,  VA 

6  Apr  87 

Department  of  the  Army  Armored  Family  of  Vehicles 

Task  Force 

Ft.  Eustis,  VA 

16  Apr  87 

U.S.  Army  Test  and  Evaluation  Command 

Aberdeen  Proving  Ground,  MD 

21  Apr  87 

U.S.  Army  Aviation  Systems  Command 

St.  Louis,  MO 

23  Apr  87 

U.S.  Army  Aviation  Center 

Ft.  Rucker,  AL 

5  May  87 

U.S.  Army  Armor  School 

Ft.  Knox,  KY 

6,7  May  87 

ARI  Field  Unit  Representatives  from  Ft.  Bliss  and 

Ft.  Huachuca 

Ft.  Bliss,  TX 
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review.  Altogether,  a  wide  range  of  approximately  120  individuals  were  contacted, 
including  TRADOC,  AMC,  and  other  independent  agencies. 

3.2.2  Conduct  of  Discussions 

Initial  contact  with  the  organizations  was  made  by  the  COR  and  visit  dates  and 
times  were  arranged.  Following  introductions  at  the  meeting  site,  a  25-minute  briefing 
was  given  by  a  member  of  the  Analytics  team  to  introduce  our  effort  and  explain  the 
purpose  of  the  meeting.  The  briefing  typically  included: 

•  Introductory  remarks 

•  What  is  OWL? 

•  The  importance  of  OWL  in  the  Army 

•  OWL  program  objectives 

•  Our  approach:  user  inputs;  matching  model;  handbooks 

•  Candidate  System  Selection  Criteria 

Upon  completion  of  the  briefing  (or  during  as  appropriate),  the  floor  was  opened 
for  questions  and  comments.  The  discussions  focused  on  whether  OWL  is  considered  in 
their  organization;  if  so,  how  it  is  considered;  what  guidance  and  tools  are  needed;  and  any 
suggestions  for  the  products  (handbook  outlines).  The  second  focus  of  discussion  was  on 
any  emerging  systems  that  the  participants  were  aware  of  that  would  be  good  candidates  for 
validation  of  operator  workload  measures.  The  selection  of  systems  is  later  described  in 
Section  5. 

3.2.3  Oue.stionnaire  Distribution 

Questionnaires  and  handbook  outlines  were  distributed  at  the  conclusion  of  each 
meeting.  Participants  were  asked  to  distribute  them  to  appropriate  individuals  within  their 
organization  and  return  the  completed  questionnaires  to  Analytics.  A  sample  questionnaire 
is  included  in  Appendix  A.  (The  handbook  outlines  are  discussed  in  Section  4.). 
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3.3  Army  Community  Concerns 


3.3.1  What  is  OWL? 


A  common  point  of  discussion  in  the  meetings  held  was  what  exactly  operator 
workload  implied.  There  seemed  to  be  an  understanding  of  the  problem  of  increasing 
technology  and  decreasing  personnel  resources  causing  increased  cognitive  operating 
requirements.  However,  there  does  appear  to  be  uncertainty  about  the  meaning  of  the  term 
"  workload."  The  diverse  concerns  included: 

•  Is  workload  based  on  the  number  of  tasks  to  be  performed? 

•  Does  the  workload  issue  revolve  around  the  scarcity  of  Cat  I  (i.e.,  high 
ability)  soldiers  and  the  necessity  of  using  Cat  IVs?  How  do  MOSs 
relate? 

•  Is  workload  physical ,  mental  or  both? 

•  How  is  maintainer  workload  related  to  operator  workload? 

•  How  does  workload  relate  to  MANPRINT?  Is  it  the  same  as 
MANPRINT? 

3.3.2  Organizational  Concerns 

The  Materiel  Acquisition  Process  consists  of  many  organizations  (both  government 
and  contractor)  performing  a  series  of  sequential  steps  to  achieve  the  goal  of  fielding 
effective  and  suitable  equipment  to  accomplish  a  mission.  The  sequential  nature  of  the 
process  assumes  the  previous  steps  have  been  adequately  accomplished  in  order  for  the 
next  steps  to  be  performed.  As  a  result  of  the  nature  of  the  MAP,  many  comments 
expressed  in  our  discussions  were  concerned  with  collection  and  use  of  the  system-relevant 
information  that  has  been  produced  previously  in  the  MAP.  A  second  major  area  of 
concern  was  resources.  The  resources  include  the  people  to  do  the  required  work,  the 
expertise  needed  to  do  the  work,  the  time  in  which  to  accomplish  the  work,  and  the  money 
with  which  to  pay  for  it.  Both  of  the  major  areas  of  concern  can  be  summed  up  in  the 
expression  that  was  heard  in  both  TRADOC  and  AMC  organizations  "TRADOC  doesn't  do 
its  job  up  front,  but  AMC  has  all  the  money."  This  expresses  the  frustrations  that:  (1) 
many  of  the  important  decisions  impacting  performance  and  OWL  (such  as  crew  size  and 
operational  capability)  are  determined  very  early  in  the  MAP  (as  in  the  requirements 
documents)  but  (2)  TRADOC  does  not  always  have  the  resources  or  information  to  make 
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knowledgeable  decisions  during  this  period.  These  decisions  and  requirements  are  the 
ones  that  will  subsequently  drive  the  design  and  development  of  the  system. 

There  seems  to  be  a  consensus  that  OWL  is  currently  addressed  in  a  reactive  mode. 
Only  after  it  has  been  identified  as  a  problem,  or  potential  problem,  is  any  action  taken. 
There  is  awareness  that  a  continuous  process  of  addressing  OWL  throughout  the  MAP 
would  enable  the  players  to  control  OWL  in  a  proactive  manner. 

The  people  involved  with  specific  systems  (e.g.  TRADOC  System  Managers 
(TSMs)  and  action  officers  in  Program  Manager  shops)  are  aware  of  potential  workload 
problems  but  don’t  have  the  guidance,  the  expertise,  an  approach,  or  the  necessary 
financial  resources  to  adequately  address  OWL.  Increasingly,  they  are  looking  toward  the 
U.S.  Army  Research  Institute  (ARI)  and  the  U.S.  Army  Human  Engineering  Laboratory 
(HEL)  for  direction  in  areas  concerning  human  performance.  The  MANPRINT  officer  is 
also  seen  as  a  resource  for  both  human  performance  and  OWL  concerns,  however,  often 
the  MANPRINT  officer  has  not  been  on  the  job  very  long  and  does  not  have  a  great  deal  of 
experience.  The  systems  people  are  looking  for  whatever  help  they  can  get,  and  often  there 
is  only  one  ARI  or  HEL  resource  person  with  more  work  than  one  person  can  realistically 
accomplish. 

Specific  comments  were  made  concerning  the  Required  Operational  Capability 
(ROC).  A  concern,  particularly  of  testers  and  evaluators,  was  that  the  ROC  does  not 
provide  sufficient  specifics  concerning  human  performance  upon  which  to  base  test  and 
evaluation  criteria.  It  is  difficult  for  them  to  know  if  there  is  a  workload  problem,  or  a 
potential  problem,  when  there  are  inadequate  measures  of  effectiveness  (MOEs)  and  no 
specified  levels  of  performance  in  the  ROC.  Certainly  there  is  some  form  of  evaluative 
judgment  involved  in  identifying  potentially  excessive  workloads,  but  there  is  frustration  in 
the  latter  portion  of  the  MAP  in  the  test  and  evaluation  area.  Those  involved  with 
evaluation  particularly  feel  that  a  good  job  is  not  being  done  in  thinking  about  the  human 
performance  issues  in  the  front  end  of  the  MAP  in  casting  the  requirements  documents. 

A  second  comment  on  the  ROC  was  that  firm  decisions  regarding  crew  size  (i.e., 
reduced  crew  size)  are  being  included,  apparently  without  front-end  analysis  being  done 
that  would  give  information  concerning  the  reduced  crew  capability.  A  comment  made  in 
one  discussion  suggested  that  the  Army  should  design  the  capability  of  the  equipment 
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around  the  capability  of  the  crew  (with  its  size,  intellectual  and  physical  abilities 
limitations)  rather  than  fitting  the  crew  to  the  operational  wish-list  of  the  equipment. 

Another  concern  was  that  the  data  expected  to  exist  do  not  always  exist.  Because  of 
the  resource  constraints  at  the  combat  developer  schools,  such  things  as  task  lists  and 
EGAs  are  not  always  developed  in  a  timely  way  (if  ever).  Concern  was  raised  that  if  any 
additional  paperwork  or  analyses  for  OWL  are  required,  that  they  probably  wouldn't  be 
done  either.  Some  felt  that  no  matter  how  useful  and  beneficial  the  analyses  might  be,  there 
will  be  some  resistance  to  doing  them  based  on  resource  constraints  as  well  as  bureaucratic 
inertia. 

3.3.3  Major  OWL  Needs 

The  major  OWL  needs  identified  from  our  meetings  with  Army  personnel  are  the 
following: 

•  The  need  for  OWL  assessment  techniques  that  will  provide  pertinent 
OWL  information  so  that  ROCs  will  provide  a  better  framework  for  aU 
subsequent  system  design  work  as  it  relates  to  human  performance 
considerations. 

•  The  need  for  OWL  assessment  techniques  that  are  not  heavily  dependent 
upon  extensive  resources  and  expertise. 

•  The  need  for  an  OWL  overview  pamphlet  that  orients  Army  personnel  to 
the  concept,  its  importance,  its  relationship  to  MANPRINT,  who  should 
be  concerned  about  it  and  when. 

3.3.4  Projected  Needs 

When  the  discussions  turned  to  anticipated  or  projected  needs  concerning 
assessment  of  operator  workload,  there  were  two  basic  categories  mentioned.  The  first  was 
the  new  emphasis  on  ASAP  and  NDI  as  the  acquisition  strategies  of  choice.  The 
streamlined  process  basically  requires  the  quality  of  development  work  to  be  done  in  a 
more  compressed  time  frame.  Therefore,  it  will  be  even  more  critical  that  the  requirements 
are  well  conceived  and  front-end  analyses  are  done  so  that  the  development  can  be 
expedited  without  running  into  design  problems.  The  NDI  strategy  presents  new  problems 
in  that  the  only  opportunity  to  ask  questions  or  obtain  data  is  in  the  market  survey  process. 
There  will  be  no  opportunity  for  testing  until  after  an  initial  purchase.  Within  the  market 
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survey,  information  can  be  requested  from  vendors,  but  there  is  no  assurance  that  the 
vendor  has  even  thought  about  the  workload  issue  or  that  any  data  pertaining  to  the  issue 
will  be  available  to  the  government.  Perhaps,  as  soldier-in-the-loop  performance  issues 
become  more  important  (e.g.,  through  emphasis  on  MANPRINT),  such  issues  and  how 
they  have  been  handled  by  the  vendors  will  be  seen  as  selling  points. 

MANPRINT  was  also  seen  as  a  driver  in  anticipated  needs  for  OWL  assessment. 
The  requirements  for  inclusion  of  MANPRINT  in  the  ROC  (AR  71-9),  the  required 
SMMP,  and  all  the  other  efforts  to  institutionalize  MANPRINT  in  the  MAP  are  seen  as 
driving  new  needs  for  data  and  analyses  concerning  soldier-in-the-loop.  The  needs  are 
particularly  seen  early  in  the  process  in  the  analytical  arena.  However,  TRADOC  plans  on 
doing  more  testing  through  its  Force  Development  Test  and  Evaluation  (FDTE)  testing 
program.  Early  testing  issues  will  become  of  more  interest  and  importance. 


3.3.5  Identified  OWL  Problems 

Our  meetings  with  Army  personnel  resulted  in  a  common  concern  being  voiced 
about  OWL.  That  is,  the  potential  for  excessive  cognitive/mental  workload  demands  being 
placed  on  operators  as  a  result  of  innovative  software  systems.  These  new  software 
systems  have  automated  many  of  the  functions  previously  done  by  operators  as  well  as 
provide  functionality  not  previously  possible  (e.g.,  new  information).  As  a  result, 
operators’  jobs  have  changed  to  being  "managers  of  information"  whereby  the 
requirements  placed  on  the  operator  are  to  manage  and  digest  the  information  provided  via 
software  systems  and  make  decisions  based  on  such  information.  This  concern  for 
excessive  cognitive/mental  OWL  was  expressed,  in  general  terms,  without  specific 
reference  to  existing  systems  per  se  but  was  foreseen  as  a  problem  with  future  developing 
systems  and  proposed  improvements  to  systems.  Section  5  in  this  report  describes  the 
prototype  systems  that  have  been  identified  as  candidates  for  the  OWL  assessment  phase 
of  this  project.  We  have  selected  prototype  systems  that  are  indicative  of  this  general 
concern  for  OWL. 
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3.3.6  Other  Workload  Initiatives 


The  issue  of  OWL  is  being  addressed  on  several  different  fronts  by  various  groups. 
These  include  Army  agencies  that  are  addressing  future  design  of  system  concepts  (Army 
HEL  Crew  Reduction  Modeling  Study)  as  well  as  theoretical  issues  concerning  OWL 
(Army  HEL  Stress  Program).  Also,  NASA  Langley  Research  Center  at  Langley,  VA  is 
exploring  OWL  in  aviation  systems  by  means  of  mental  states  estimation.  We  have  made 
contact  with  these  groups  and  have  exchanged  ideas  concerning  our  effort  as  well  as  theirs. 
Below  are  brief  descriptions  of  these  programs  objectives  as  well  as  their  stage  of 
progress. 

We  will  continue  to  identify  other  OWL  initiatives  so  to  keep  abreast  of  the.  latest 
developments  in  the  field  and  evaluate  the  applicability  of  these  initiatives  to  our  program. 

•  Army  HEL:  Crew  Reduction  Modeling  Study,  Aberdeen  Proving 
Ground,  MD.  HEL  is  the  beginning  stages  of  developing  a  computer- 
driven  mathematical  model  to  evaluate  the  feasibility,  desirability,  and 
effectiveness  of  reducing  the  size  of  combat  vehicle  crews  (e.g.,  tanks). 

They  are  in  the  process  of  selecting  an  outside  contractor  for  computer 
modeling  of  combat  crew  operation  (Sept  87). 

•  Army  HEL:  Combat  Stress  Program,  Aberdeen  Proving  Ground,  MD. 

A  multi-disciplinary  research  program  has  been  started  to  provide  basic 
human  and  weapon  systems  performance  data  under  "combat-stress" 
conditions  by  means  of  modeling  the  battlefield  conditions  that  soldiers 
are  subjected  to.  This  is  a  4-phase  program  that  is  presently  in  Phase  I. 

In  Phase  I,  efforts  are  underway  to  determine  which  "stress  indices"  are 
likely  to  reflect  the  effects  of  acute  exposure  to  stress  (i.e.,  battlefield 
conditions)  in  test  participants  and  are  likely  to  work  best  in  simulating 
such  conditions  in  the  laboratory. 

•  NASA  Langley  Research  Center:  Mental  States  Program,  Langley,  VA. 

NASA  has  just  started  a  5  year  program  whose  initial  objectives  are  to 
identify  physiological  indices  that  reflect  "mental  states"  such  as  fatigue 
and  boredom  which  have  been  shown  to  influence  operator 
performance.  Ultimately,  these  physiological  signs  will  be  monitored 
by  sophisticated  software  systems  during  aircraft  flights  in  order  to 
identify  the  points  in  time  when  software  intervention  is  needed  for 
maintaining  ^ght  performance. 


3.4  Questionnaire  Results 


Nineteen  people  have  fully  responded  to  the  survey  questionnaire  to  date.  Though 
this  is  a  relatively  small  sample,  the  data  provide  a  basis  for  discussion  concerning  the 
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Army  needs  for  guidance  on  OWL.  A  copy  of  the  survey  questionnaire  can  be  found  in 
Appendix  A. 

3.4.1  Demographics 

Nine  respondents  were  civilians  who  worked  for  Army  organizations  such  as 
TRADOC,  ARI  and  HEL  while  the  remaining  ten  respondents  were  military  personnel 
who  performed  various  roles  in  the  MAP,  (e.g.,  TRADOC  assistant  system  manager). 

With  respect  to  functional  roles,  eight  respondents  indicated  they  have  MANPRINT 
responsibilities/functions  that  support  TRADOC  schools  (n=2),  TRADOC  integrating 
centers  (n=2),  and  test  and  evaluation  agencies  (n=4).  The  remaining  eleven  respondents 
were  TRADOC  project  officers  (n=2),  TRADOC  assistant  system  managers  (n=2),  and 
representatives  from  ARI  (n=3),  HEL  (n=l)  and  TRADOC  integrating  centers  (n=3). 

The  nineteen  respondents  as  a  group  represent  integral  players  in  the  MAP  who  can 
contribute  toward  addressing  OWL  during  system  development.  The  major  organization 
not  adequately  represented  in  this  sample  was  materiel  developers  (AMC).  Our  future 
plans  are  to  include  an  adequate  sample  from  AMC. 

3.4.2  Operator  Workload  (OWLl 

The  questionnaire  contained  specific  questions  on  the  importance  of  OWL  with 
respect  to  respondents'  work  (present  and  future),  the  means  they  use  now  to  address 
OWL  and  future  directions  (guidance)  needed  to  address  OWL.  The  results  were 
somewhat  svuprising. 

When  asked  how  often  the  issue  of  OWL  should  be  considered  in  their  work,  a 
majority  of  respondents  (n=13)  stated  "often"  (based  on  a  4  choice  category  scale  -  "often", 
"sometimes",  "rarely"  or  "never").  In  addition,  respondents  foresaw  a  greater  emphasis 
on  addressing  OWL  in  their  work  for  several  reasons.  Almost  all  respondents  (n=17)  saw 
changes  in  requirements  (i.e.,  MANPRINT)  as  a  force  in  directing  their  future  efforts 
toward  addressing  OWL.  Also,  a  large  majority  (n=16)  saw  "changes  in  technology" 
with  respect  to  system  innovations  as  an  important  factor  in  directing  their  work  efforts  to 
addressing  OWL.  Based  on  such  an  overwhelming  consensus  that  OWL  is  an  important 


issue  to  be  addressed  in  their  work,  one  would  anticipate  respondents  would  employ 
similar  methodologies/tools  to  address  workload;  this  was  not  the  case.  When 
respondents  were  asked  to  state  the  specific  guidance  (documents)  they  use  to  address 
OWL,  eleven  (11)  people  stated  they  have  no  source  document  for  addressing  OWL  issues. 
The  remaining  eight  (8)  respondents  gave  assorted  answers  such  as  EGA,  HARDMAN, 
OWL  studies  in  journals,  and  operator  task  lists  when  available. 

When  respondents  were  asked  how  they  would  like  to  address  OWL,  answers 
varied  between  respondents.  For  example,  some  offered  no  suggestions  (n=5).  Other 
respondents  stated  that  specific  organizations  should  address  OWL  but  not  "my" 
organization  (n=3).  The  remaining  eleven  respondents  gave  individual  answers  such  as 
more  resources  devoted  to  OWL  issues,  better  defined  ROC  documents  with  respect  to 
human  performance,  videotaping  operators  performing  task  scenarios,  task  analysis, 
objective  performance  measures,  and  physiological  measures. 

When  respondents  were  asked  to  state  the  guidance  they  would  like  to  have  for 
addressing  OWL,  several  suggestions  were  offered.  These  were: 

•  Standardized  methodology/tool  for  addressing  OWL  that  requires 
minimal  resources,  it  is  non-intrusive,  in  real  time  and  characterized  as 
objective  in  nature  (n=4) 

•  Training  course  on  what  OWL  is  and  how  to  address  it  (n=2) 

•  Local  points  of  contact  (POC)  for  OWL  (n=2) 

•  OWL  Handbook  for  writing  ROC  documents  (n=l) 

•  How  to  raise  funds  to  address  OWL  (n=l). 

•  Preparation  of  a  MIL-H-46855  Workload  DID  (n=l) 

Based  on  these  findings,  it  seems  apparent  that  Army  personnel  are  concerned 
about  addressing  OWL  in  their  work,  however,  there  seems  to  be  a  lack  of  uniform 
direction  and  understanding  with  regard  to  what  OWL  is,  how  to  address  it,  and  where 
might  one  find  answers  to  these  questions. 

3.4.3  Respondents'  Work  Responsibiliries  during  the  MAP 

The  survey  contained  a  series  of  questions  to  profile  the  work  done  by  Army 
personnel  as  it  relates  to  the  MAP.  Questions  pertained  to  identifying  major  work 
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responsibilities,  the  inputs  and  outputs  to  such  work  as  well  as  the  sources  of  information 
used  to  accomplish  this  work. 

Table  3.4.3-1  lists  the  major  areas  of  responsibilities  stated  by  respondents. 
Almost  half  of  the  respondents  (n=9)  indicated  having  several  overlapping  responsibilities 
during  the  MAP,  (i.e.,  they  responded  to  more  than  one  category). 


Table  3.4.3-1  Responsibilities/roles  during  the  MAP  that  are  held  by  respondents 

Responsibility/Role _ ^ _ Number  of  Respondents 

Define  or  review  requirements,  10 

standards,  criteria 

Develop  or  monitor  the  design  8 

of  emerging  system  concepts 

Design  or  monitor  the  characteristics  7 

of  early  prototype  systems 

Test  &  evaluation  of  systems  (early,  7 

mid-term,  late) 

MANPRINT  (R&D)  2 


As  seen  in  Table  3.4.3-1,  most  respondents'  responsibilities  are  centered  in  the  early 
portions  of  the  MAP.  Their  roles  are  seen  as  critical  for  identifying  OWL  issues  early  in 
the  MAP  such  that  OWL  can  be  addressed  in  a  proactive  mode.  This  is  further  exemplified 
by  the  fact  that  the  recipients  of  their  work  are  integral  players  in  the  MAP.  Table  3  4.3-2 
shows  the  major  organizations  and  functional  roles  who  are  the  recipients  of  the  work 
accomplished  by  the  survey  respondents.  Some  respondents  indicated  having  several 
recipients  of  their  work. 
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Table  3A.3-2  The  major  organizations  and  functional  roles  who  are  the  recipients  of  the 
work  produced  by  survey  respondents 

Organizations/Functional  Roles _  Number  of  Respondents 

TRADOC  Schools  (e.g.,  combat  developers)  7 

Army  Materiel  Command  (e.g.,  materiel  developers)  7 
Test  &  Evaluation  Organizations  (e.g.,  OTEA)  5 

TRADOC  Headquarters  2 


In  summary,  the  respondent's  responsibilities  and  their  associated  work  are  an 
integral  part  of  the  MAP  and  serve  to  direct  all  future  work  that  occurs  during  system 
development.  But,  the  respondents  lack  standard  methodologies/tools  to  address  OWL 
issues  as  seen  by  their  answers  to  questions  about  OWL. 

3.4.4  Sources  for  Information 


With  respect  to  fulfilling  their  job  responsibilities,  respondents  listed  the  Army 
documents  as  well  as  agencies  that  they  referred  to  or  seek  guidance.  We  were  interested  in 
identifying  these  sources  in  order  to  understand  the  types  of  information  and  guidance 
sought  by  respondents.  Such  information  would  provide  insights  to  the  work  issues  that 
respondents  felt  were  important  but  lack  the  knowledge  or  experience  to  address  these 
issues  solely  by  themselves.  Table  3.4.4-1  lists  the  major  sources  of  such  guidance.  Most 
respondents  used  multiple  sources. 
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Table  3.4. 4-1  Major  sources  for  information  and  guidance  that  are  used  by  respondents  to 
fulfill  their  job  responsibilities 

Organizations _ Number  of  Respondents 

Army  Research  Institute  (ARI)  11 

Army  Human  Engineering  Laboratory  (HEL)  7 

TRADOC  Headquarters  6 

TRADOC  Schools  (e.g.,  TRADOC  system  manager)  4 

Directorate  of  Combat  Development  3 

Test  &  Evaluation  Organizations  (e.g.,  OTEA)  2 


Documents _ Number  of  Respondents 


DOD  Directive 
5000.3 

Test  &  Evaluation 

4 

AR  602-2 

MANPRINT 

11 

AR70-10 

Test  &  Evaluation 

5 

AR71-3 

User  Testing 

4 

AR71-9 

Materiel  Objectives  &  Requirements 

4 

AR70-8 

Personnel  Performance  &  Training 

2 

MIL  STD 

Human  Engin.  Design  Criteria  for 

3 

1472-C 

Military  Systems,  Equipment,  & 

Facilities 

HARDMAN 

(Hardware  vs.  Manpower  Compar.  Anal.)l 

ECA 

(Early  Comparability  Analysis) 

1 

Clearly,  the  respondents'  sources  for  information  (e.g.,  ARI,  MANPRINT)  reflect 
their  concern  to  address  human-related  issues  (i.e.,  human  performance).  It  is  of  interest 
to  note  that  the  documentation  sought  for  guidance  contains  minimal  information  on  OWL. 
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3.4.5  Performance  Issues  with  respect  to  the  Total  System  Development  Process 


The  survey  contained  a  series  of  questions  to  ascertain  the  different  performance 
areas  that  respondents  consider  in  order  to  do  their  job.  Table  3.4.5- 1  summarizes 
participants'  responses  to  four  major  performance  areas.  Listed  are  the  number  of 
respondents  who  responded  to  a  4-choice  category  scale  ("often",  "sometimes",  "rarely", 
or  "never")  by  checking  the  "often'  category  with  respect  to  these  performance  areas. 
Table  3.4.5-2  summarizes  participants'  responses  to  major  human  performance  areas. 
Listed  are  the  number  of  respondents  who  responded  to  a  4-choice  category  scale  ("often", 
"sometimes",  "rarely",  or  "never")  by  checking  the  "often"  category  with  respect  to  these 
performance  areas. 

Table  3.4.5-1  Number  of  respondents  who  stated  that  they  "often"  consider  these 
performance  issues  in  their  work 

Performance  Area _ Number  of  Respondents 


Total  System  Performance 

16 

Subsystem  Performance 

10 

Operator  Performance 

16 

Maintainer  Performance 

13 

Table  3.4.5-2  Number  of  respondents  who  stated  that  they  "often"  consider  these  human 
performance  areas  in  their  work 

Human  Performance  Area _ Number  of  Respondents 


Human  Factors  Engineering 

14 

Manpower 

11 

Personnel 

13 

Training:  Individual  soldiers 

16 

Training:  Unit 

8 

Safety 

11 

Health  Hazards 

10 
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It  is  quite  evident  from  these  results  (Table  3.4.5-1,  &  Table  3.4.5-2)  that 
performance  issues  (total  system  and  human  element  areas)  are  given  consideration  by 
respondents. 


3.4.6  Conclusions 

The  results  of  the  study  show  that  respondents  are  aware  and  concerned  about 
human  performance  areas.  However,  a  standardize  methodology/tool  as  well  as  a  single 
source  for  information  addressing  OWL  is  lacking.  It  also  seems  apparent  that 
respondents'  lack  of  clarity  in  their  answers  to  specific  questions  about  OWL  (e.g.,  no 
responses)  indicate  a  fuz?iness  on  what  is  meant  by  OWL  and  how  to  address  it.  This 
finding  is  very  revealing  as  the  respondents  to  this  survey  have  ideal  positions  from  which 
to  address  OWL  throughout  the  MAP. 


3.5  User  Suggestions  for  OWL  Program  in  Army 


Within  the  context  of  the  discussions  with  users,  a  number  of  suggestions  were 
given  as  to  what  the  users  thought  would  be  most  useful  to  Army  users.  Although  most  of 
the  suggestions  have  already  been  touched  upon,  a  separate  listing  was  thought  to  be 
helpful.  The  items  are  not  in  any  specific  order.  The  suggestions  are: 

•  Integrate  with  the  MANPRINT  effort  to  ensure  success. 

It  was  recognized  by  many  with  whom  we  spoke  that  to  assure 
implementation  of  any  OWL  guidance  that  was  developed,  some  type 
of  regulatory  emphasis  was  needed.  The  most  practical  suggestion  was 
that  the  MANPRINT  requirements  (e.g.,  SMMP)  be  used  as  the 
vehicles  to  address  OWL  concerns. 

•  Guidance  must  accommodate  limited  resources  and  expertise  available. 

•  Make  the  cognitive/mental  aspects  of  OWL  the  explicit  focus  of  the 
project. 

•  Capitalize  on  any  existing  information  or  data  that  relates  to  OWL. 

Information  is  generated  and  analyses  performed  to  make  decisions  in 
the  current  MAP.  Before  attempting  to  require  more  data  and 
information  generation,  be  sure  to  examine  what  is  already  available  to 
see  if  it  might  be  useful  in  answering  OWL  questions. 

•  Both  TRADOC  and  AMC  must  be  receptive  to  this  effort.  To  get  the 
most  benefit,  both  must  be  involved  in  OWL  assessment  and  control. 
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Create  an  associated  OWL  DID. 


There  is  an  awareness  that  contractors  will  be  in  the  correct  place  to 
gather  data  relevant  to  OWL.  In  order  to  capture  those  data,  as  well  as 
assuring  that  an  appropriate  methodology  is  being  followed,  create  a 
Data  Item  Description  so  that  there  is  a  means  to  obtain  data  from  the 
contractor  during  system  design  and  development. 

•  Have  awareness  that  contractors  may  be  ones  to  use  the  developed 
OWL  guidance. 

Early  analyses  that  requires  technical  expertise  or  more  man-hours  that 
internally  available  may  be  contracted  out.  Similarly,  the  realization  that 
the  RFP  is  the  most  appropriate  place  to  require  0\^  assessment. 


3.6  User  Suggestions  for  OWL  Products 


The  users  made  several  suggestions  regarding  the  handbooks  to  be  produced.  The 
suggestions  are: 

•  Computer-based  mode  of  presentation. 

The  users  questioned  the  use  of  written  material  (i.e.,  handbooks). 

Their  experience  has  been  that  there  is  a  tendency  to  just  put  handbooks 
on  a  shelf  and  not  use  them.  The  suggestion  was  that  guidance, 
particularly  the  predictive  and  evaluative  handbooks,  created  on  an  on¬ 
line,  interactive,  computer-based  system  might  be  better  and  easier  to 
use. 

•  Pamphlet  should  be  used  to  institutionalize  the  concept  of  OWL. 

Many  integral  players  in  the  MAP  do  not  consider  soldier-in-the-loop 
when  developing  system  concepts  and  designs.  The  pamphlet  may  be 
useful  to  draw  attention  to  the  soldiers'  performance  aspects  of  system 
performance  and  to  advise  managers  to  raise  flags  when  an  OWL 
problem  may  be  involved. 

•  Two  recommendations  should  result  from  our  guidance  for  selecting 
specific  OWL  techniques. 

Recommendations:  1)  a  minimum  assessment  battery  for  low-resource 
applications  ,  and  2)  a  more  complete  battery  of  OA^  techniques  for 
circumstances  where  more  resources  are  available. 


3.7  Conclusions  and  Recommendations 


Several  conclusions  can  be  drawn  from  the  discussions  that  were  held: 
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•  Within  the  Anny  community,  there  is  real  concern  about  OWL  but  there 
seems  to  be  lack  of  conformity  in  how  to  address  it.  Everyone  seems  to 
have  their  own  way  of  doing  it  or  ignoring  it 

•  There  was  skepticism  of  any  project  as  wide-reaching  (i.e.,  going 
across  organizational  boundaries)  as  this  one. 

•  It  is  important  for  us  to  keep  abreast  of  other  Army  initiatives 
concerning  operator  workload. 

•  Any  program  or  methodology  developed  has  to  be  sensitive  to  resource 
limitations. 

•  The  methodology  must  be  able  to  accommodate  the  ASAP,  NDI,  and 
other  acquisition  strategy  procedures. 


3.8  Follow-on  Interview  Plans 


Those  individuals  in  Army  organizations  with  whom  we  spoke  provided  useful 
information.  We  will  continue  our  discussions  with  them  throughout  the  project  as 
appropriate  for  later  follow-ups.  Current  plans  are  to  keep  them  apprised  of  the  project 
progress  and  to  consult  with  them  again  in  the  future  concerning  the  handbooks.  Since 
our  primary  goal  is  to  provide  useful  information  in  the  most  appropriate  format,  the  Army 
community  of  users  will  be  consulted  regarding  the  development  of  the  handbooks. 
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4.  OUTLINE  OF  FINAL  PRODUCTS 


4.1  INTRODUCTION 


In  the  original  statement  of  work  (SOW),  five  major  areas  were  identified  to  be 
addressed  by  a  set  of  handbooks  which  Army  personnel  would  use  for  making  decisions 
on  OWL  during  the  materiel  acquisition  process  (MAP).  These  areas  were: 

•  How  and  where  to  use  handbooks 

•  Guidelines  for  how  to  estimate  OWL,  including  how  to  select  the  most 
appropriate  measures  of  OWL  for  a  given  new  system  design 

•  Guidelines  for  evaluating  OWL  during  concept  and  evaluation,  and 
developmental  and  operational  testing 

•  Alternative  methods  for  reducing  excessive  OWL  in  Army  systems,  and 

•  Recommendations  for  selecting  among  or  prioritizing  those  alternative 
OWL  reduction  methods. 

These  five  major  areas  were  originally  proposed  to  be  covered  by  three  related 
products:  an  overview  pamphlet  for  the  TRADOC  community  on  OWL,  and  two 
handbooks  addressing  OWL  techniques,  one  to  be  used  during  the  early  phases  of  the 
MAP  (pre-Milestone  1  activities)  and  the  other  to  be  used  during  system  development 
phases  of  the  MAP  (post-Milestone  1  activities). 

To  ensure  these  products  are  successfully  received  by  the  Army  community,  we 
developed  draft  outlines  depicting  each  product's  content  as  well  as  descriptions  of  the 
rationale  for  each  section  covered.  These  outlines  were  used  to  elicit  Army  personnel 
comments  and  reactions  in  order  to  ascertain  their  specific  needs  concerning  OWL  and  to 
"shape"  the  final  handbook  products.  It  was  the  first  step  in  our  ongoing  and  iterative 
development  of  user-oriented  products.  Examples  of  the  draft  outlines  that  were  discussed 
in  the  interviews  with  Army  personnel  may  be  found  in  Appendices  B  through  D. 

Based  on  meetings  with  Army  personnel,  we  identified  two  major  concerns 
needed  to  be  addressed  in  all  the  OWL  products  to  ensure  their  successful  use 
incorporation  into  the  work  activities  of  the  intended  users  of  these  products. 

•  A  thorough  description  of  what  operator  workload  is,  the  importance  of 
OWL  as  a  significant  determinant  of  system  performance,  and  how  it 
relates  to  system  requirements  and  design  issues. 


that 

and 
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•  A  descriptive  framework  to  show  the  integration  between  existing  Army 
requirements  (e.g.,  MANPRINT  requirements)  and  other  Army 
programs  (e.g.,  HARDMAN)  and  the  approach  prescribed  in  the 
handbooks.  How  the  OWL  program  complements  and  supports  these 
-  existing  Army  programs. 

Besides  these  overall  considerations,  each  outline  elicited  specific  comments  and 
suggestions.  These  user  reactions  will  be  discussed  in  the  respective  subsections  for 
each  proposed  product. 


4.2  TRADOC  PAMPHLET 


4.2. 1  Original  Concept  in  Statement  of  Work  CSOWl 

The  TRADOC  overview  pamphlet  was  originally  conceived  as  a  guide  for 
TRADOC  personnel  "to  incorporate  workload  in  the  ROC  (Required  Operational 
Capability)".  This  pamphlet  was  seen  as  an  overview  that  described  what  is  meant  by 
OWL  and  how  to  address  it  in  a  ROC  .  It  would  highlight  the  issues  of  OWL  such  that 
TRADOC  managers,  (i.e.,  combat  developers),  could  recognize  the  need  to  address  OWL 
in  ROC  documents  and  assist  them  to  make  provisions  (e.g.,  requirements)  for  its  adequate 
assessments  in  subsequent  phases  of  the  MAP. 


4.2.2  Rationale  for  Original  Draft  Outline  and  Revision 


Our  original  draft  outline  expanded  the  scope  of  the  pamphlet.  We  envisioned  the 
pamphlet  providing  an  overview  emphasizing  the  need  to  address  OWL  throughout  the 
MAP  such  that  managers  in  TRADOC  and  AMC  who  are  in  positions  to  oversee  the 
development  of  systems  would  have  the  proper  framework  to  address  OWL  throughout  the 
cycle.  Otherwise,  issues  that  relate  to  OWL  could  be  overlooked  as  the  MAP  progresses. 
Such  an  emphasis  on  OWL  throughout  the  MAP  would  provide  the  correct  orientation  for 
OWL  to  be  addressed  in  a  proactive  mode  as  opposed  to  a  reactive  mode  of  fixing  past 
mistakes  attributed  to  excessive  operator  workload.  The  original  pamphlet  outline  dated 
9Feb87  is  found  in  Appendix  B. 

Based  on  our  discussions  with  Army  personnel  who  are  involved  with  various 
phases  of  the  MAP,  it  became  apparent  that  the  pamphlet  should  go  beyond  our  original 
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conceptualization  with  respect  to  our  intended  audience.  That  is,  there  seemed  to  be  a  lack 
of  awareness  and/or  concern  that  OWL  is  an  important  issue  to  be  addressed  throughout  the 
MAP. 

As  a  result,  the  pamphlet  needs  to  address  OWL  such  that  all  integral  players  in  the 
MAP  (e.g.,  TRADOC,  AMC,  OTEA,  AMSAA,  and  TECOM)  are  aware  of  the  importance 
of  addressing  OWL  throughout  the  MAP  since  all  can  contribute  to  preventing  OWL 
problems.  To  do  so  required  revising  our  initial  pamphlet  outline  such  that  ALL  key 
personnel  in  the  MAP  would  be  oriented  to  conceptualizing,  developing  and  evaluating 
systems  with  OWL  as  a  major  consideration.  This  is  especially  true  today  since  new 
technologies  (automation  via  software  innovations)  and  projected  manpower  reductions  are 
placing  a  potentially  heavier  burden  on  operators  to  mentally  perform  new  operations  that 
can  directly  impact  system  performance.  We  have  revised  the  manager's  pamphlet  outline 
to  reflect  this  orientation  toward  OWL  so  ALL  Army  personnel  involved  in  system 
development  realize  the  coordinated  effort  needed  across  organizations  to  ensure  that  OWL 
is  addressed  in  a  proactive  mode.  The  revised  pamphlet  outline  dated  23May87  follows 
this  subsection. 
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DATE:  23MAY87 


REVISED  MANAGER’S  OPERATOR  WORKLOAD  ASSESSMENT  PAMPHLET 


OUTLINE 

USER  PROFILE:  The  intended  users  for  the  pamphlet  are  the  managers  who  are  involved 
in  both  delineating  the  needs,  and  developing  the  requirements  for  a  new  system  as  well  as 
those  involved  in  evaluating  system  performance  in  which  the  soldier-in-the-loop  is 
considered  an  integral  part  of  the  evaluation.  This  user  is  not  interested  in  the  details  of 
workload  estimation  or  evaluation.  What  IS  of  interest  to  these  managers  is  high-level 
guidance  on  what  are  the  Army  requirements  regarding  workload,  and  what  high-level 
provisions  should  be  built  into  the  system  acquisition  strategy  for  the  assessment  of  OWL. 
The  orientation  of  this  pamphlet  is  to  instill  the  concept  of  operator  workload  (OWL)  in  the 
everyday  vocabulary  of  managers  such  that  it  is  addressed  in  a  proactive  mode  throughout 
the  MAP.  This  can  only  happened  if  ALL  managers,  irrespective  of  organizations,  are 
attuned  to  the  importance  of  OWL  as  a  major  factor  contributing  to  overall  system 
performance.  Each  manager  has  a  role  in  ensuring  that  OWL  does  not  adversely  affect 
overall  system  performance. 

FORMAT:  This  Pamphlet  will  be  structured  to  provide  a  concise,  easily  understood 
presentation  of  the  role  of  OWL  control  in  the  materiel  acquisition  process  (MAP).  Tables, 
charts,  flow  diagrams,  and  specific  examples  will  be  used  liberally  to  promote  quick 
apprehension  of  concepts. 

GOAL:  Provide  the  reader  with  an  overview  of  the  role  of  OWL  control  in  the  materiel 
acquisition  process,  including  the  nature  of  the  problem,  DoD/DA  documents  and 
requirements  concerning  OWL  control,  and  available  technologies  to  assist  ALL  Army 
managers  (e.g.,  TRADOC,  AMC,  OTEA,  AMSAA,  and  TECOM)  in  ensining  OWL 
control.  Provide  guidance  in  accessing  other  OWL  control  resources,  especially  the  OWL 
Prediction  and  Evaluation  Handbooks. 
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LENGTH:  approximately  40-50  pages 
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CONTENTS 


I.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B .  Traditional  factors  attributed  to  OWL,  e.g.,  physical  involvement  of  operators 

C.  New  technologies  affecting  system  concepts,  e.g.,  automation  via  software 

D.  New  factors  attributed  to  OWL,  e.g.,  mental/cognitive  involvement  of  operators 

E.  Impact  of  OWL  on  Army  Mission  Functions 

F.  Army  requirements,  specifications,  standards  and  regulations  for  OWL 

G.  Relationship  of  OWL  to  MANPRINT  Program 

H.  Contribution  of  ALL  managers  involved  with  the  MAP  in  addressing  OWL 

I.  Description  of  the  contents  of  this  pamphlet,  how  to  use  this  pamphlet 

STRATEGY:  Introduce  managers  to  the  key  OWL  concepts  and  regulations.  Provide  the 
proper  framework  on  how  to  use  this  handbook. 


II.  OVERVIEW  OF  OWL  FUNDAMENTALS 

A.  OWL  Performance  Model  -  factors  contributing  to  workload. 

B .  OWL  with  various  types  of  actions/behaviors  that  are  mental/cognitive  in  nature 

1 .  Searching  for  and  receiving  information 

2.  Identifying  objects,  actions,  events 

3.  Problem  solving 

4.  Decision  making 
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C.  OWL  considerations  for  system  types  with  respect  to  Mission  Areas 

D.  OWL  considerations  during  the  Materiel  Acquisition  Process  (MAP) 

1 .  Accelerated  Systems  Acquisition  Process  (ASAP) 

2.  Proactive  mode  vs.  reactive  mode  in  addressing  OWL 

3.  Key  questions  concerning  OWL  to  be  addressed  throughout  the  MAP 

E.  OWL  Assessment  Program 

1.  Prediction  (analytic  approach) 

2.  Evaluation  (empirical  approach) 

3.  Analysis  of  results  -  addressing  key  OWL  questions  as  revealed  by 
analysis  of  one's  results/data 

F.  OWL  Control  Plan 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  key  areas  and  steps 
involved  in  OWL  prediction/evaluation  and  its  relationship  to  the  materiel  acquisition 
process. 


III.  OWL  IN  REQUIREMENTS  ANALYSIS/CONCEPT  FORMULATION 

A.  TRADOC  perspective  -  combat  developers 

B .  AMC  perspective  -  program  managers 

C.  Evaluators/testers  perspective 

D.  OWL  trade  offs  in  concept  formulation 

E.  Development  of  a  preliminary  OWL  Control  Plan 

F.  Key  OWL  resources 

1.  Documents 

2.  Organizations  (e.g,  HEL,  ARI) 

3.  Individuals  (e.g.,  HFE  specialists) 

G  The  TRADOC  Manager's  OWL  Concept  Formulation  Check  List 
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1.  What  should  the  TRADOC  combat  developer  be  ensuring  is 
accomplished. 

H.  The  AMC  Manager's  OWL  Concept  Formulation  Check  List 

1.  What  should  the  AMC  program  manager  be  ensuring  is  accomplished. 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  requirements  analysis  and  concept  formulation.  Provide  the 
manager  the  knowledge  to  integrate  the  OWL  Control  Plan  with  existing  requirements, 
programs,  and  other  control  plans  that  are  part  of  the  materiel  acquisition  process. 


IV.  OWL  IN  SYSTEM  DEVELOPMENT 

A.  TRADOC  perspective  -  system  managers 

B.  AMC  perspective  -  program  managers 

C.  Evaluators/testers  perspective 

D.  The  OWL  Control  Plan 

E.  Methods  for  assessment 

1 .  OWL  Prediction  (analytic  approach) 

2.  OWL  Evaluation  (empirical  approach) 

F.  Key  OWL  resources 

1.  Documents 

2.  Organizations  (e.g.,  HEL,  ARI) 

3.  Individuals  (e.g.,  HFE  specialists) 

G.  TRADOC  system  manager's  OWL  Check  List  for  OWL  Prediction 

H.  TRADOC  system  manager's  OWL  Check  List  for  OWL  Evaluation 

I.  AMC  program  manager's  OWL  Check  List  for  OWL  Prediction 

J.  AMC  program  manager's  OWL  Check  List  for  OWL  Evaluation 
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K.  Testers/Evaluators  Check  List  for  OWL  Prediction 

L.  Testers/Evaluators  Check  List  for  OWL  Evaluation 

STRATEGY:  Provide  the  manager  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  Assessment  Program  with  existing  requirements,  programs,  and  other 
control  plans  that  are  part  of  the  materiel  acquisition  process. 


V.  ITERATIVE  NATURE  OF  OWL  ASSESSMENT 

A.  Materiel  acquisition  process 

B.  System  design  decisions 

C.  Evolution  of  OWL  considerations 

STRATEGY:  Establish  the  concept  that  the  OWL  Assessment  Program  and  its  management 
and  control  are  evolving  processes  which  are  modified  as  the  materiel  acquisition  process 
progresses  and  requires  the  coordination  and  cooperation  of  all  Army  agencies  involved  in 
the  MAP. 


VI.  OWL  CONCERNS  AND  ARMY  SYSTEM  DEVELOPMENT  ITEMS 

A.  Non-Developmental  Items  (NDI) 

B.  Product  Improvements  (P3I,  PIP) 

STRATEGY:  Elaborate  on  the  special  circumstances  these  areas  present  for  addressing 
OWL  and  emphasis  the  need  to  ensure  that  OWL  does  present  itself  as  a  problem. 
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VIL  EXAMPLE 


A.  An  example  will  be  provided  that  delineates  the  various  managerial  responsibilities 
across  organizations  that  play  a  role  in  controlling  OWL  during  the  MAP  -  the 
development  and  implementation  of  their  respective  OWL  Control  Plans. 

STRATEGY:  Provide  the  user  with  a  concrete  example  of  an  overall  OWL  Control  Plan. 


VII.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY :  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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4.3  PREDTCnON  HANDBOOK 


4.3.1  Original  Concept  in  Statement  of  Work  ('SOW') 

The  Workload  Prediction  Handbook  was  originally  conceived  as  a  guide  for 
addressing  OWL  during  the  new  system  concept  development  phase  of  the  MAP  (pre- 
Milestone  I  activities).  The  handbook  would  direct  Army  personnel  in  employing  OWL 
predictive  techniques  by  identifying  the  most  appropriate  predictive  technique  with  respect 
to  their  particular  needs  and  resources.  The  techniques  offered  would  lend  themselves  to 
identifying  OWL  issues  that  need  to  explore  and/or  address  in  Ae  further  refinement  of  the 
system  concept.  Use  of  such  techniques  would  give  valuable  direction  early-on  in  the 
MAP  so  to  minimize  potential  OWL  problems  in  later  phases  of  the  MAP. 

4.3.2  Rationale  for  Original  Draft  Outline  and  Revision 

In  general,  our  original  draft  outline  attempted  to  provide  a  methodology  for 
identifying  the  most  appropriate  predictive  technique  for  a  given  system  concept.  It  also 
highlighted  the  importance  of  OWL,  its  relationship  to  system  performance  and  OWL 
evaluations  conducted  during  later  phases  of  the  MAP.  The  original  pamphlet  outline 
dated  9Feb87  is  to  be  found  in  Appendix  C. 

Based  on  our  meetings  with  Army  personnel,  we  identified  sections  in  the  outline 
that  needed  further  clarification  as  well  as  new  material  to  be  included  in  the  handbook. 
The  predictive  handbook  needs  to  address  and/or  clarify  the  following: 

•  Greater  emphasis  on  the  significance  of  addressing  OWL  early-on  in  the 
MAP  -  greater  likelihood  of  success,  (i.e.,  incorporating  OWL 
considerations  in  later  design  phases),  as  well  as  the  most  cost-effective 
point  in  time  to  offer  suggestions  for  design  changes  (i.e.,  least  expense 
involved  in  comparison  to  later  phases  of  the  MAP). 

•  The  complementary  relationship  between  the  use  of  this  handbook  and 
the  Army's  new  directive  to  address  soldier  issues  early-on  in  the  MAP 
-  MANPRINT  Program. 

•  The  relationship  between  the  guidance  offered  in  this  handbook  and  the 
methodologies  currently  used  by  the  Army  during  the  early  portions  of 
the  MAP,  (e.g.,  HARDMAN  and  EGA). 

•  How  the  results  generated  from  the  use  of  these  predictive  techniques 
can  drive  the  later  phases  of  the  MAP  in  controlling  OWL.  What  are  the 
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means  available  to  ensure  this,  (i.e.,  ROC  ,  future  test  plans,  DIDs, 
etc.). 

We  have  revised  the  predictive  handbook  outline  to  reflect  these  changes.  The 
revised  predictive  handbook  outline  dated  23May87  follows  this  subsection. 


DATE:  23MAY87 


REVISED  WORKLOAD  PREDICTION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Prediction  Handbook  is  intended  for  Army  personnel 
(e.g.,  TRADOC  combat  developer  and  system  manager)  during  the  concept  and'-early 
design  phases  of  the  materiel  acquisition  process  (MAP).  This  user  is  interested  in  the 
different  OWL  measures  and  techniques  applicable  during  early  design.  This  user  is 
typically  the  person  who  (1)  makes  the  decision  of  which  OWL  assessment  tools  to  use, 
and  (2)  adapts  those  tools  to  fit  the  specific  needs  and  characteristics  for  the  system  of 
interest.  To  guide  the  user  in  performing  these  functions,  the  handbook  will  identify 
(based  on  system  requirements  and  specific  design  objectives)  the  workload  assessment 
methodology  needed  via  a  "matching  model"  procedure  such  that  an  optimal  OWL 
Assessment  Battery  is  offered.  The  handbook  provides  guidance  on  the  identification  and 
implementation  of  the  OWL  Assessment  Battery  for  all  types  of  systems. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
user  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an  OWL 
assessment  program  during  early  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  assessment  program  with  existing  requirements,  programs  (e.g., 
MANPRINT) ,  and  methodologies  (e.g.,  EGA  and  HARDMAN)  that  are  used  in  the  early 
phases  of  the  materiel  acquisition  process  (MAP). 

LENGTH:  approximately  75-100  pages 
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CONTENTS 


I.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B.  Traditional  factors  attributed  to  OWL,  e.g.,  physical  involvement  of  operators 

C.  New  technologies  and  factors  affecting  system  concepts,  (e.g.,  automation  via 
software) 

D.  New  factors  attributed  to  OWL,  (e.g.,  mental/cognitive  involvement  of  operators) 

E.  Impact  of  OWL  on  overall  system  performance 

F.  Army  requirements,  specifications,  standards,  and  regulations  for  OWL 

G.  Relationship  of  OWL  to  MANPRINT  Program 

H.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  operator 
workload  assessment  program  via  OWL  predictive  techniques  during  concept  and 
preliminary  design  phases 

I.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  key  OWL  concepts  and  requirements.  Orient  the  user  to 
the  critical  concept  -  OWL  predictive  assessment  techniques  are  critical  for  identifying 
OWL  issues  such  that  solutions  concerning  OWL  (overload)  can  be  reached  in  a  cost- 
effective  way  (proactive  mode).  Otherwise,  potential  OWL  problems  will  probably  be 
addressed  in  a  reactive  mode,  (i.e.,  fixing  past  mistakes).  Provide  the  framework  on  how 
to  use  the  Handbook. 
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II.  OVERVIEW  OF  OPERATOR  WORKLOAD  PREDICTION 

A.  What  is  OWL  prediction? 

B .  Relationship  between  methodology  offered  in  this  document  and  existing  Army 
methodologies,  (i.e.,  ECA,  HARDMAN,  Task  Analysis) 

C.  The  relationship  between  OWL  prediction  &  OWL  evaluation 

D.  Factors  to  consider  for  OWL  prediction 

1.  System  requirements 

2.  Operator  capabilities/skills/behaviors  required 

3.  OWL  performance  model  -  performance  factors  to  consider 

4.  OWL  assessment  techniques 

E.  Methodology:  Matching  model  for  establishing  an  OWL  Predictive  Assessment 
Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
prediction  (analytic  approach)  and  its  relationship  to  OWL  evaluation  (empirical  approach) 


III.  OPERATOR  WORKLOAD  PREDICTION  METHODOLOGY 


A.  Determine  system  performance  requirements 
1 .  Potential  Sources: 

a.  O  &  O  Plans  (Operational  &  Organizational) 

b.  JMSNS  (Justification  of  Major  Systems  New  Starts) 

c.  ECA  (Early  Comparability  Analysis) 

B .  Determine  operator  actions/behaviors  for  system  usage 
1.  Sources 

a.  Requirement  documents  (O  &  O  Plans,  ROCs) 

b.  Task  analyses 

c.  Expert  opinions  (SMEs) 
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d.  Comparative  systems  (e.g.,  SMMP,  EGA) 

e.  Mock-ups 

f.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system  usage 
(cf.,  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell,  & 
Shearer,  1964) 

-  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 

c.  Problem  solving 

d.  Decision  making 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
methodology  (decision  making  process) 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  predictive  assessment  battery 

2.  Decision  tables  for  OWL  methodologies  that  are  applicable 

F.  Analytic  assessment  methods 

1 .  Purpose:  Prediction  of  OWL  &  "chokepoints" 

2.  Sensitivity 

3.  Representation  of  OWL  issues/behaviors 

4.  Techniques 

a.  Expert  opinions 

b.  Comparisons 

c.  Simulations/Models 

d.  Math  models 

e.  Task  analytic  methods 

5.  Interpretation  of  assessment  results 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an 
OWL  Predictive  Assessment  Program.  Provide  the  user  the  know-how  to  integrate  the 
OWL  Assessment  Program  with  existing  requirements,  programs,  and  methodologies  that 
are  part  of  the  materiel  acquisition  process. 


IV.  ITERATIVE  NATURE  OF  OWL  PREDICnON 


A.  Materiel  acquisition  process 

B.  System  design  decisions 

C.  How  to  address  OWL  issues  in  the  MAP,  (i.e.,  ROC,  SMMP,  and  Test  Plans) 

STRATEGY:  Establish  the  concept  that  the  OWL  assessment  program  (developed  from 
using  this  Handbook)  is  an  evolving  program  which  will  be  modified  as  the  materiel 
acquisition  process  progresses.  OWL  must  be  monitored  and  controlled  by  establishing 
requirements,  (i.e..  Measures  of  Effectiveness  [MOEs]  in  the  ROC),  that  ensure  activities 
conducted  after  Milestone  I  address  OWL  issues  that  were  identified  from  the  use  of  the 
OWL  predictive  techniques. 


V.  EXAMPLES 

A.  Examples  will  be  provided  for  each  of  the  assessment  techniques. 

STRATEGY:  Provide  the  user  with  concrete  examples  of  OWL  predictive  assessment 
programs.  Provide  reality  to  the  OWL  prediction  methodology  described  in  the  Handbook. 


VI.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials.  , 
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4.4  EVALUATION  HANDBOOK 


4.4. 1  Ori^nal  Concept  in  Statement  of  Work  ( SOW^ 

The  Workload  Evaluation  Handbook  was  originally  conceived  as  a  guide 
for  addressing  OWL  during  the  conduct  of  concept  evaluation,  and  developmental  and 
operational  testing  (post-Milestone  I  activities).  The  handbook  would  direct  Army 
personnel  in  employing  OWL  evaluation  techniques  (subjective,  physiological,  and 
objective/task  measures)  by  identifying  the  most  appropriate  empirical  technique  with 
respect  to  their  particular  needs  and  resources.  The  techniques  offered  would  lend 
themselves  to  be  incorporated  in  any  system  evaluation  effort  and  would  complement  any 
existing  data/information  on  OWL  collected  earlier  in  the  MAP  (predictive  techniques). 
These  techniques  provide  data  to  substantiate  design  solutions/decisions  to  minimize  OWL 
as  well  as  direct  future  refinements  for  system  design. 


4.4.2  Rationale  for  Original  Draft  Outline  and  Revision 


In  general,  our  original  draft  outline  attempted  to  provide  a  methodology 
for  identifying  the  most  appropriate  evaluative  technique  for  a  given  system  during  the 
various  developmental  phases  phases  after  Milestone  I  activities.  It  also  highlighted  the 
importance  of  OWL,  its  relationship  to  system  performance  and  previous  OWL 
assessments  conducted  earlier  during  the  MAP  (OWL  predictive  techniques).  The  original 
handbook  outline  dated  9Feb87  is  to  be  found  in  Appendix  D. 


Based  on  our  meetings  with  Army  personnel,  we  identified  Army  concerns  that 
need  to  be  addressed  in  the  Evaluation  Handbook.  These  concerns  are  the  following: 

•  Emphasis  on  the  coordination  of  previous  OWL  assessments  (OWL 
predictive  techniques)  with  subsequent  testing  and  evaluation  such  that 
provisions  are  established  (e.g.,  ROC,  SMMP)  to  ensure  that  OWL 
concerns  already  identified  are  addressed  adequately  in  the  later  phases 
of  the  MAP. 

•  Elaboration  on  the  key  roles  that  the  various  testing  and  evaluation 
agencies  (OTEA,  AMSAA,  and  TECOM)  as  well  as  system  and 
training  agencies  (AMC  and  TRADOC)  play  in  addressing  OWL  issues. 

All  have  a  contribution  to  make  in  controlling  OWL. 

•  Address  the  applicability  of  the  methodology  proposed  in  the  handbook 
with  respect  to  the  Army’s  accelerated  programs  for  system  acquisition. 
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(i.e.,  ASAP  and  NDI),  and  the  evolutionary  development  of  systems, 

(i.e.,  PIP  and  P3I). 

We  have  revised  the  evaluation  handbook  to  reflect  these  concerns.  The  revised 
evaluation  handbook  outline  dated  23May87  follows  this  subsection. 
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DATE:  23MAY87 


REVISED  WORKLOAD  EVALUATION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Evaluation  Handbook  is  intended  primarily  for  the 
Army  community  involved  with  developmental  testing  and  evaluation  (DT&E).  These 
users  are  interested  in  interpreting  the  results  of  the  various  workload  assessments 
conducted  throughout  the  development  cycle  and  are  primarily  concerned  with  system 
evaluation  from  many  different  perspectives  (e.g.,  TRADOC,  AMC  ,  OTEA,  AMSAA, 
and  TECOM).  These  evaluators  and  users  of  data  from  test  and  evaluation  (T&E)  are  also 
interested  in  how  to  perform  OWL  analysis.  In  addition,  they  are  concerned  with  the  actual 
constraints  placed  by  real-world  implementation  of  a  workload  assessment.  Such 
constraints  involve  traditional  (and  non-traditional)  limits  on  testbed  resources  (e.g., 
subjects,  time,  and  funding).  The  T&E  users  are  also  concerned  with  how  to  transform 
OWL  data  and  information  into  recommendations  for  system  design. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
reader  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing 
an  OWL  assessment  program.  Provide  the  user  the  know-how  to  integrate  the  OWL 
assessment  program  with  existing  requirements,  programs,  and  methodologies  that  are  part 
of  the  military  system  acquisition  cycle. 

LENGTH:  approximately  125-150  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B.  Traditional  factors  attributed  to  OWL,  (e.g.,  physical  involvement  of  operators) 

C.  New  technologies  and  factors  affecting  system  concepts,  (e.g.,  automation  via 
software) 

D.  New  factors  attributed  to  OWL,  (e.g.,  mental/cognitive  involvement  of  operators) 

E.  Impact  of  OWL  on  overall  system  performance 

F.  Army  requirements,  specifications,  standards,  and  regulations  for  OWL 

G.  Relationship  of  OWL  to  MANPRINT  Program 

H.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  operator 
workload  assessment  program  via  OWL  evaluative  techniques  during  concept 
evaluation  and  developmental  and  operational  testing. 

I.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  the  key  OWL  concepts  and  requirements.  Orient  the 
user  to  the  critical  concept  -  OWL  Evaluative  Techniques  are  crucial  for  determining  the 
feasibility  of  design  decisions  as  they  relate  to  minimizing  OWL.  Provide  the  proper 
framework  on  how  to  use  the  handbook. 


11.  OVERVIEW  OF  OPERATOR  WORKLOAD  EVALUATION 
A.  What  is  OWL  evaluation? 
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B. 


Who  is  involved?  (TRADOC,  AMC,  OTEA,  AMSAA,  TECOM, ) 


B .  The  relationship  between  OWL  evaluation  &  OWL  prediction 

C.  Factors  to  consider  for  OWL  evaluation 

1 .  System  requirements 

2 .  OWL  assessment  program  constraints 

a.  Subjects 

b.  Time 

c.  Funding 

d.  Etc. 

3.  Operator  capabUities/skills/behaviors  required 

4.  Earlier  OWL  assessments  (OWL  prediction) 

5.  OWL  performance  model  -  performance  factors  to  consider 

6.  OWL  empirical  assessment  techniques 

D.  Methodology:  Matching  model  for  establishing  an  OWL  Evaluative  Assessment 
Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
evaluation  (empirical  approach)  and  its  relationship  to  OWL  prediction  (analytic  approach), 
and  system  design. 


III.  OWL  ASSESSMENT  (TEST  AND  EVALUATION) 

A.  Determine  system  performance  requirements 
1.  Sources 

a.  Requirement  documents  (ROC,  and  O  &  O  Plans) 

B .  Determine  operator  actions/behaviors  for  system  usage 

1,  Sources 

a.  Task  analyses 

b.  Expert  opinions 

c.  Comparative  systems 

d.  Mock-ups 
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e.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system  usage 
(cf..  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell,  & 
Shearer,  1964)  -  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 

c.  Problem  solving 

d.  Decisionmaking 

C.  Determine  stage  in  materiel  acquisition  process 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
(decision  making  process) 

1.  For  example,  environmental  factors  such  as  noise,  vibration,  heat,  and 
cold 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  Assessment  battery 

2.  Decision  tables  for  workload  methodologies  that  are  applicable 

F .  Establish  framework  for  the  evaluation 
1.  Task  scenarios 

G.  Empirical  Assessment  Methods 

1 .  Purpose:  Evaluate  design  decisions 

2.  Sensitivity 

3.  Task  scenarios:  representation 

4.  Techniques 

a.  Operator  opinion 

b.  Primary  task 

c.  Secondary  task 

d.  Physiological  responses 

5.  Interpretation  of  assessment  results  to  impact  system  design 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  implementing  an  OWL  evaluative 
assessment  program  -  test  and  evaluation.  Provide  the  user  the  know-how  to  integrate  the 
OWL  assessment  program  with  existing  requirements,  programs,  and  methodologies  that 
are  part  of  the  materiel  acquisition  process. 
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IV.  ITERATIVE  NATURE  OF  OWL  EVALUATION 


A.  Materiel  acquisition  process  (MAP)  and  Accelerated  acquisition  process  (ASAP) 

B.  System  design  decisions 

C.  Army  agencies  involved  in  the  iterative  nature  of  OWL  assessment  -  (TRADOC  and 
AMC  as  key  players ) 

STRATEGY:  Establish  the  concept  that  OWL  assessment  (test  and  evaluation)  is  an 
iterative  process  that  is  conducted  throughout  the  materiel  acquisition  process  to  ensure 
OWL  is  not  a  system  problem. 


V.  OWL  CONCERNS  AND  ARMY  SYSTEM  DEVELOPMENT  ITEMS 

A.  Non-Developmental  Items  (NDI) 

B.  Product  Improvements  (e.g.,  P3I,  PIP) 

STRATEGY:  Elaborate  on  the  special  circumstances  these  areas  present  for  addressing 
OWL  and  show  the  applicability  of  the  methodology  represented  in  this  handbook  to  these 
set  of  circumstances. 


VI.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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5.  SELECTION  OF  REPRESENTATIVE  SYSTEMS 


5.1  Introducrion 


The  three  representative  system  selected  during  the  present  effort  are  crucial  to  later 
efforts.  They  provide  the  test-bed  upon  which  the  validation  of  OWL  measures  and 
methodology  will  be  later  conducted  (Task  4).  In  addition,  these  also  provide  the 
framework  for  illustration  of  applications  which  will  augment  of  the  documentation  of  our 
methods  (OWL  Handbooks)  which  will  be  subsequendy  be  prepared  (Task  5).  Beyond 
thishowever,  it  is  intended  that  the  analyses  of  the  prototype  systems  will  provide 
significant  benefits  for  their  development  and  the  Army.  Delineated  in  the  following  are: 
the  selection  methodology  (5.2),  description  of  the  selected  systems  (5.3),  and  brief 
discussion  of  on-going  efforts  (5.4). 


5.2  Methodology 

A  two-stage  approach  was  applied  for  selection  of  representative  systems  based 
upon  Interview  Inputs  and  Army  Sponsor  Guidance. 

5.2.1  Interview  Inputs 

The  first  stage  involved  identification  of  candidate  systems  during  our  interviews  of 
potential  users  and  cognizant  personnel  within  the  Army  (cf.,  Sect.3.2).  Typically 
subsequent  to  description  of  the  OWL  Assessment  Program,  interviewees  were  presented 
with  candidate  system  selection  criteria.  Most  salient  of  these  were  requirements  to:  (1) 
select  candidates  requiring  the  broad  range  of  predictive  and  evaluative  OWL  techniques 
and  (2)  select  systems  which  could  be  realistically  impacted  within  the  time  frame  of  the 
program  efforts.  The  first  of  these,  it  is  noteworthy,  was  reflected  in  interest  with  systems 
in  different  phases  of  the  materiel  acquisition  process,  and  under  different  combat 
developers  consequently  representing  a  range  of  systems.The  second  requirement  pointed 
toward  systems  undering  rapid  (NDI)  or  near-term  changes  in  status.  The  interviews 
proved  fruitful  and  pointed  toward  more  than  a  dozen  individual  or  families  (e.g.,  AFV)  of 
systems  for  joint  consideration  with  ARI. 
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5.2.2  Army  Sponsor  Guidance 


Candidates  systems  were  weighed  with  respective  to  theobjectives  and  schedule  of 
the  present  effort  in  a  joint  meeting  6-7  May  at  the  ARI  Field  Unit,  Ft.  Bliss.  Based  upon 
information  obtained  in  interview  follow-ups,  system  candidates  were  first  screened  with 
respect  to  the  program  schedules  of  the  various  systems  as  well  as  assessability  by  the 
evaluation  team.  For  surviving  candidates,  operators  of  interest  were  identified  and 
evaluated.  Group  evaluations  were  made  with  respect  to  operator  gross  environmental 
exposure  (impact);  estimated  relative  levels  of  perceptual,  cognitive,  motor,  and 
communication  workload;  as  well  as  physiological  workload  levels  imposed  by  manual 
tasks  (lifting,  pushing,  carrying  etc.).  Representative  systems  were  then  selected  so  as  to 
insure  a  range  of  perceptual,  cognitive,  motor,  and  communication  workload  levels  across 
systems. 


5.3  Selected  Representative  Systems 

The  selected  representative  systems  include:  (1)  Line  of  Sight-Forward  (Heavy) 
[LOS-F(H)];  (2)  Automatic  Target  Handoff  System  [ATHS];  as  well  as  (3)  a  Remotely 
Piloted  Vehicle  (RPV[AQUILA  or  lEW-UAV(I)].  In  the  following,  each  of  these  systems 
will  initially  be  characterized  with  regard  to  nature,  type  of  acquisition  and  status,  as  well  as 
operators  of  interest.  The  comparison  of  the  character  of  anticipated  OWL  will 
subsequently  be  overviewed  across  systems  for  the  operators  of  interest. 

5.3.1  LOS-Frm 

LOS-F(H)  is  one  of  five  components  of  the  Forward  Area  Air  Defence  (FAAD) 
System.  FAAD  as  a  whole  is  designed  to  defend  force  and  critical  assets  against  rotary 
wing  and  fixed  wing  aircraft  threats  during  24  hour  day  and  night  operations,  a 
countermeasures  environment,  and  adverse  visibility  and  weather  conditions.  The  LOS- 
F(H)  component  of  FAAD  will  be  a  self  propelled,  armored,  highly  mobile,  platform  with 
a  primary  armament  of  launch-ready  missiles  as  well  as  a  complementary  weapon 
providing  full  coverage  of  air  defence  within  the  dead  zone  of  the  missile.  It  is  designed  to 
provide  front  line  air  defence  against  attacks  by  high  performance  ground  attack  aircraft, 
attack  helicopters,  as  well  as  self  defence  against  armored  vehicles  and  ground  targets. 


5-2 


LOS-F(H)  is  being  acquired  under  a  NDI  strategy  aimed  at  fielding  during  1991  which 
provides  for  evaluation  of  the  breadth  of  measures  of  OWL.  Candidate  selection. 
Milestone  I/II  of  the  ASAP,  is  scheduled  to  be  made  on  the  basis  of  a  competitive  shoot-off 
to  be  conducted  during  September  to  December  1987.  Follow-on  testing  of  the  selected 
candidate  as  will  as  Force  Development  Test  and  Evaluation  (FDTE)is  scheduled  for  Fiscal 
Year  (FY)  88.  LOS-F(H)  has  two  operators  of  particular  interest  with  respect  to  workload 
that  may  be  nominally  designated  across  system  candidates  as:  (1)  Gunner  and  (2)  Squad 
Leader. 

5.3.2  Automatic  Target  Handover  System  FATHSI 

The  ATHS  is  designed  to  provide  secure  jam  resistant  digital  communications  and 
is  envisioned  as  being  a  subsystem  in  a  variety  of  future  helicopter  and  other  vehicles. 
With  associated  Navigation  and  Interrogation  Friend  or  Foe  (IFF)  Subsystems,  it  is 
currently  scheduled  for  integration  into  the  APACHE  AH-64A  Aircraft  (the  draft  RFP  is 
under  review).  This  integration  is  for  facilitation  of  intelligent  automatic  digital  data 
processing  for  missiles,  weapons,  and  related  information  transfer  between,  other  AH- 
64A,  Scout  Aircraft,  and  ground  units  via  digital  waveforms  compatible  with  TACFIRE. 
For  the  AH-64A,  the  integration  is  to  allow  simultaneous  operation  by  dual  [Pilot  and 
Copilot/Gunner  (CPG)]  control  and  display  units  which  shall  be  night-vision  goggle 
compatible.  It  is  believed,  however,  that  in  flight  the  primary  operator  will  be  the  CPG 
because  of  the  flight  control  responsibilities  of  the  Pilot.  The  CPG  will  have  a  message 
received  indicator  "that  can  be  viewed/heard  during  all  workload  conditions".  The  CPG  is 
the  operator  which  has  been  identified  for  detailed  workload  evaluation. 

5.3.3  Aqiiila/  TRW-TTAVm  Remotely  Piloted  Vehicles  fRPVsl 

The  Aquila  and  Unmanned  Air  Vehicle  -  Intelligence  Electronic  Warfare  (Interim) 
[lEW-UAV(I)]  are  RPV  components  of  representative  systems  pointed  to  as  having 
growing  significance  for  the  Army.  Of  these,  the  Aquila  based  system  was  judged 
somewhat  more  suitable  for  the  goals  of  the  present  (OWL)  effort  because  of  its 
developmental  maturity  and  history.  However  as  it  is  under  Full  Scale  Engineering 
Development  (FSED),  with  an  impending  scheduled  ASARC  in  June  1987,  and  it  was 
deemed  prudent  to  retain  as  a  backup  the  lEW-UAV(I).  Currently  with  broad  opportunities 
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for  the  present  (OWL)  efforts  because  of  its  status  as  a  NDI,  the  lEW-UAV(I)  is  being 
selected  on  the  basis  of  an  on-going  flyoff  and  will  be  used  for  development  of  the 
specifications  of  final  UAVs  by  Ft.  Huechuca.  Based  upon  initial  analyses  and  the  status 
of  the  lEW-UAV(I),  it  was  judged  that  for  the  present  purposes  it  could  be  considered 
analogous  to  (a  larger)  Aquila  (with  similar  perceptual,  cognitive,  motor,  and 
communication  performance  and  workload  problems).  Consequently,  Aquila  will  be 
delineated  in  the  following  both  with  regard  to  itself  and  as  a  surrogate  for  the  lEW- 
UAV(I). 

The  Target  Acquisition/Designation  and  Aerial  Reconnaissance  System  (TADARS) 
Aquila  RPV  is  an  'eye  in  the  sky  system'  designed  to  provide  the  ground  commander 
realtime  battlefield  information  by  detection,  recognizing,  identifying  and  designating 
enemy  forces  (ARI,  1987).  The  Acquila  itself  is  a  tailless  mid-wing  tactical  mini-plane 
with  a  rear-mounted  pusher  propeller  engine.  Included  as  part  of  the  system  are  a  stabilized 
TV  sensor  and  a  laser  rangefinder/designator  on  the  aircraft,  an  antijam  data  link  for 
communication  with  a  ground  control  station,  as  well  as  personnel  for  operation.  Basically 
fixed  during  operations,  the  ground  control  station  may  be  noted  as  being  located  in  an 
protected  shelter  in  the  rear  of  a  MS  14  5-Ton  Truck.  More  interestingly,  the  data  link  may 
be  noted  as  having  has  seven  (0-6)  levels  of  function  which  may  impact  on 
mission/modes(Search,  Artillery  and  Track)  performance  and  workload.  Although  more 
recent  flight  tests  have  implications  regarding  this  of  which  we  are  not  fully  apprised  (e.g., 
GAO,  1986;  ARI,  1987,  p.4),  Hershberger  et  al.,  1983  have  previously  reported 
suggestive  results  from  "man  in  the  loop"  simulations.  Aquila  has  three  operators  of  which 
two  have  been  identified  for  detailed  workload  evaluation: 

(1)  Vehicle  Operator  (VO) ,  and 

(2)  Mission  Commander  (MC). 

5.3.4  OWL  Overview  For  The  Selected  Systems 

Figmre  5.3.4-1  summarizes  salient  features  of  operator  workload  developed  during 
the  group  evaluation  meeting  6-7  May  at  Ft.  Bliss  (cf.,  5.2.2).  Examining  this  table,  it 
may  be  seen  that  except  for  the  Gunner  in  the  LOS-F(H)  (during  loading  operations), 
physiological  workload  is  expected  to  relatively  constant  and  minimal  (Low).  The  group 
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judgements  of  perceptual,  cognitive,  motor,  and  communication  workload  levels  may  also 
be  seen  to  span  the  range  (Low-High). 


5.4  Discussion 

The  three  selected  representative  systems  are  the  test-beds  upon  which  the 
validation  of  OWL  measures  and  methodology  will  be  later  conducted  (Task  4).  Although 
a  large  step,  their  selection  is  only  the  beginning  of  the  development  of  an  overall  and 
system  specific  plans  for  the  methodological  validation  be  developed  during  the  next  phase 
of  efforts  (Task  3).  To  insure  the  success  of  the  efforts,  as  well  as  provide  for  the 
significant  benefit  for  the  representative  systems  through  OWL  analysis,  will  require  long 
and  close  collaboration  with  system  and  other  cognizant  personnel.  In  anticipation  of  this, 
personnel  were  identified  during  our  interviews  of  users  and  other  selected  personnel 
within  the  Army  (cf.,  Sect.3.2).  Cognizant  personnel  are  currently  being  contacted 
regarding  the  selection  of  systems  and  coordination  of  joint  efforts  in  beginning. 
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Figure  5.3.4- 1  OWL  Overview  for  the  selected  systems 


6.  OTHER  PROGRAM  PROGRESS 


Noteworthy  progress  has  been  toward  program  goals  which  are  tangential  to  the 
subject  of  the  present  report  (re;  Task  2).  OWL  information  collection  and  analysis  has 
been  and  is  underway  in  preparation  for  the  evaluation  of  workload  measures  which  is  to 
be  conducted  as  part  of  the  next  phase  of  effort  (i.e.,  Task  3).  For  the  interested  reader, 
details  of  the  status,  nature,  and  methods  of  this  collection  may  be  found  delineated  in  the 
appendix  describing  the  OWL  Information  System.  The  OWL  information  analysis  effort, 
it  is  pertinent  to  observe,  is  based  upon  a  taxonomy  of  workload  assessment  methods  with 
two  broad  classes: 

♦  Analytic  -  predictive  techniques  that  may  be  applied  earliest  in  system 
design  before  "man  in  the  loop"  studies;  and 

•  Empirical  -  operator  workload  assessments  that  are  taken  during 
simulator,  prototype,  or  system  evaluations. 

Analytic  techniques  have  initially  received  our  greatest  interest  and  attention.  This 
is  partially  because  (although  not  in  the  context  of  the  system  evaluation  objectives  of  the 
Army):  (1)  the  empirical  techniques  have  received  considerable  attention,  (e.g.,  O'Donnell 
and  Eggemeier,  1986);  and  (2)  a  model  matching  empirical  techniques  with  user 
requirements  has  been  been  reported  (Casper,  et  al.,1986)  although  apparently  currently 
unavailable.  Our  motivation  for  focusing  on  the  analytic  techniques  lies  in  their  application 
during  the  earliest  developmental  stage  where  the  greatest  design  flexibility  is  available  at 
the  least  cost  as  pointed  out  earlier  in  this  report  (cf..  Sect.  2).  Thus  far,  we  have  classified 
these  analytic  techniques  into  five  categories: 

♦  Comparability  Analysis  —  This  category  involves  the  application  of 
existing  workload  data  from  an  comparable  earlier  system  to  estimate  the 
workload  for  the  system  under  development  (e.g.,  Shaffer,  et  al., 

1986). 

*  Mathematical  Models  —  Various  mathematical  techniques  have  been 
used  in  a  theoretical  context  for  a  long  time.  Transfer  functions, 
information  theory,  and  queuing  theory  are  some  of  the  techniques  of 
this  category  (e.g..  Senders,  1964).  These  provide  basic  limits  or 
boundaries  which  may  be  applied  during  "front  end  analysis"  with 
regard  to  OWL. 

•  Expert  Opinion  —  The  category  relies  on  the  opinion  of  experts  who 
have  intimate  working  knowledge  of  the  mission  objectives,  the 
operational  environment  and  the  workload  of  comparable  systems  . 

This  category,  in  contrast  to  listed  below,  relies  upon  experts  to  identify 
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choke-points  and  does  not  require  formal  task  analyses  (Zachary, 

1981). 

•  Task  Analytic  Techniques  --  This  category  involves  the  development  of 
-  a  mission  profile  which  represents  the  way  the  system  is  to  be  used.  A 

number  of  investigators  have  used  the  approach  (e.g.,  Stone,  Gulick, 
and  Gabriel,  1985).  The  profile  makes  it  possible  to  perform  a  task 
analysis  for  a  given  work  station  and  translate  it  into  activity  profiles  as 
a  function  of  time.  This  procedure  will  uncover  major  situations  in 
which  the  time  available  to  perform  the  task  (mission)  exceeds  the  time 
available. 

•  Simulation  Models  —  These  attempt  to  model  human  behavior  and 
thereby  predict  performance.  Some  operate  at  the  level  of  operator  tasks 
(task  analysis)  to  predict  level  of  performance  (e.g.  Siegel  and  Wolf, 

1969).  Others  extend  the  task  analysis  approach  and  carry  the  scenario 
into  far  more  detail;  macro  and  micro  models  of  components  of  the  task  . 
are  used  to  build  accuracy  and  time-line  projections  of  human 
performance  (e.g.,  Harris,  et  al.,  1986). 

We  are  presently  documenting  a  procedure  for  the  selection  from  among  these 
analytic  categories  based  on  user  requirements  and  resources.  As  an  initial  step,  this 
procedure  distinguishes  OWL  comparability  analyses  from  other  categories  by  the 
requirement  for  comparable  system  data.  In  subsequent  steps  going  from  Mathematical 
Models  to  Simulation  Models,  the  other  categories  are  distinguished  by  their  increasing 
requirements  for  formal  system  and  operator  task  definitions.  Our  plan  is  to  shortly 
complete  an  overview  of  our  procedure  for  selection  among  analytic  assessment  of  OWL 
(Hill,  et  al.,  in  preparation). 
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7.  FUTURE  PLANS 


7. 1  Future  Plans  for  Subsequent  Tasks 

Task  3  will  consist  of  a  critical  review  and  evaluation  of  OWL  measurement 
techniques  (predictive  and  evaluative)  and  the  development  of  validation  plans  for  the  OWL 
methodology  to  be  applied  to  the  three  Army  prototype  systems. 

Validation  plans  will  be  prepared  to  include  OWL  measures  that  offered  the  greatest 
utility  and  impact  on  the  prototype  systems  described  earlier  [LOS-F(H),  ATHS,  and 
Aquila/IBW-UAV(I)].  For  each  system  ,  a  goal  of  the  validation  plans  will  be  to  utilize 
families  of  OWL  techniques.  For  example,  a  family  of  predictive  techniques  (e.g.,  expert 
opinion,  simulation  models)  will  be  applied  to  systems  in  order  to  assess  their  relative 
utility  in  providing  similar  types  of  information  before  man-in-the-loop  simulations. 
Similarly,  a  family  of  evaluative  techniques  (e.g.,  subjective  techniques  such  as  SWAT, 
TLX,  and  Modified  Cooper-Harper  Scales)  will  be  applied  to  systems  in  order  to  assess 
their  relative  utility  in  providing  information  during  man-in-the-loop  evaluations.  The 
validation  plans  will  be  structured  to  be  reflective  of  crew  workload  issues  to  the  extent 
possible  within  the  validation  effects.  OWL  analyses  will  also  be  directed  at  ensuring 
benefit  for  the  selected  systems. 

In  Task  4,  the  plans  generated  in  Task  3  will  be  implemented.  We  will  conduct 
studies  to  validate  the  OWL  measures  and  methodology  while  performing  OWL  analyses 
on  the  three  selected  systems.  In  addition  to  demonstrating  and  validating  the  OWL 
methodology,  it  is  intended  that  the  analyses  of  the  prototype  systems  will  provide 
significant  and  direct  benefits  for  the  development  of  these  systems.  This  is  extremely 
important  since  the  success  of  this  OWL  program  will  be  direct  function  of  the  Army 
community's  reaction  to  our  approach.  The  more  we  can  document  our  "successes"  in 
using  this  methodology,  the  greater  the  likelihood  that  the  Army  community  will  be 
receptive  to  using  our  methodology  (products). 

Task  5  will  address  the  production  of  the  primary  documentation  of  the  OWL 
program.  We  will  produce  three  products:  Manager's  OWL  Pamphlet,  OWL  Prediction 
Handbook  and  an  OWL  Evaluation  Handbook.  A  Post-Contract  Survey  form  will  be 
prepared  to  provide  an  efficient  vehicle  to  assess  the  degree  to  which  these  OWL 
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documents  have  met  their  goals  for  the  Army.  The  technical  report  detailing  the  scientific 
basis  for  the  information  contained  in  the  pamphlet  and  handbooks,  and  discussing  further 
research  in  the  area  of  controlling  operator  workload  will  be  prepared  as  the  final  product 
for  the  Army. 
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Appendix  A 


OPERATOR  WORKLOAD  (OWL)  SURVEY 
ARI  CONTRACT  NO.  MDA903-86-C-0384 


PLEASE  RETURN  TO: 
ANALYTICS  INC.  (  ATTN:  OWL) 
2500  MARYLAND  ROAD 
WILLOW  GROVE,  PA.  19090 
(215)  657-4100  x.164 


PG.2 


RESPQNSIBILITIES/RQLES  IN  THE  MATERIEL  ACQUISITION  PROCESS  (MAR 

1.  PLEASE  INDICATE  YOUR  ROLE(S)  IN  THE  MAP. 

(CHECK  APPROPRIATE  ITEM(S). ) 


DEFINE  OR  REVIEW  REQUIREMENTS,  STANDARDS,  CRITERIA 

DEVELOP  OR  MONITOR  THE  DESIGN  OF  EMERGING  SYSTEM  CONCEPTS 

DESIGN  OR  MONITOR  THE  CHARACTERISTICS  OF  EARLY  PROTOTYPE 
SYSTEMS 


TEST  AND  EVALUATION  OF  SYSTEMS  (EARLY,  MID-TERM,  LATE) 
DURING  MAP. 


OTHER  (PLEASE  SPECIFY: 


) 


2.  FOR  OR  TO  WHOM  DO  YOU  RESPOND  -  WHOSE  TASKS/DIRECTIVES  DO  YOU  USE 
TO  DO  YOUR  WORK?  (CHECK  APPROPRIATE  ITEMS) 

_ 0PM  (OFFICE  PROGRAM  MANAGER) 

_  TSM  (TRADOC  SYSTEM  MANAGER) 

_  DCD  (DIRECTORATE  OF  COMBAT  DEVELOPMENT) 

_  DOTD  (DIRECTORATE  OF  TRAINING  DEVELOPMENT) 

_  T&E  DIV/BD  (TEST  &  EVALUATION) 

_  OTHER  (PLEASE  SPECIFY: _ 

_ ) 
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PG3 

3.  WHAT  GUIDANCE  AND  ASSISTANCE  DO  YOU  USE  IN  FULFILLING  YOUR 
RESPONSIBILITY?  (CHECK  APPROPRIATE  SOURCES  AND  SPECIFY) 

A.  DOCUMENTS: 

_  DOD  # _ 

_  AR  # _ 

_  FM  # _ 

_  TM  # _ 

_  OTHER  DOCUMENTATION  (PLEASE  SPECIFY: _ 

_ ) 

B.  AGENCIES:  PLEASE  SPECIFY  (e.g.,  HEL,  ARI...) 


4.  TYPICALLY,  WHO  USES  THE  OUTPUT  OF  YOUR  EFFORTS  AND  PRODUCTS  ? 
(PLEASE  SPECIFY) 


PG.4 


5.  HOW  OFTEN  DO  YOU  CONSIDER  THE  FOLLOWING  PERFORMANCE  AREAS 
IN  FULFILLING  YOUR  JOB  RESPONSIBILITIES? 


6.  HOW  OFTEN  DO  YOU  CONSIDER  THE  FOLLOWING  HUMAN  PERFORMANCE  AREAS 
IN  FULFILLING  YOUR  JOB  RESPONSIBILITY? 

(CHECK  IN  APPROPRIATE  BOXES) OFTEN  SOMETIMES  RARELY  NEVER 

HUMAN  FACTORS 
ENGINEEERING 

MANPOWER 

PERSONNEL _ 

TRAINING  : _ 

INDIVIDUAL  SOLDIERS 

UNIT _ _ 

SAFETY 

HEALTH  HAZARDS 


OTHER:  PLEASE  SPECIFY 
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OPERATOR/  MAINTAINER  WORKLOAD 

7.  DOES  THE  ISSUE  OF  OPERATOR  AND  MAINTAINER  WORKLOAD  (OWL)  LEVEL  GET 
CONSIDERED  IN  YOUR  WORK  ? 


OFTEN 

SOMETIMES 

RARELY 

NEVER 

IF  EVER,  HOW  DO  YOU  ADDRESS  OWL?  (e.g.,  SPECIFIC  TOOLS,  EDUCATED  GUESSES) 


8.  WHAT  SPECIFIC  GUIDANCE  OR  DOCUMENTS  DO  YOU  NOW  USE  TO  ADDRESS  OWL? 
e.g.,  ARS,  LOCAL  REGs,  SOPs. 


9.  HOW  OFTEN  SHOULD  THE  ISSUE  OF  WORKLOAD  LEVELS  BE  CONSIDERED  IN  YOUR 
JOB? 


OFTEN 

SOMETIMES 

RARELY 

NEVER 
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10.  HOW  WOULD  YOU  LIKE  TO  ADDRESS  OWL  ? 


1 1 .  WHAT  GUIDANCE  WOULD  YOU  LIKE  TO  HAVE  FOR  ADDRESSING  OWL?  (e.g.  POC, 
DOCUMENT...) 


12.  DO  YOU  FORESEE  AN  INCREASED  CONCERN  WITH  OWL  DUE  TO; 

(CHECK  APPROPRIATE  ITEM{S)) 

_  CHANGES  IN  TECHNOLOGY 

_  CHANGES  IN  REQUIREMENTS 

_  OTHER  (PLEASE  SPECIFY: _ ) 

_  NONE 
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BACKGROUND  INFORMATION 


YOUR  TITLE  : _ GRADE  /  RANK _ 

POSITION: _ 

YRS.  IN  CURRENT  POSITION: _ 

AGENCY  /  UNIT:  _ 

SYSTEMS  INVOLVED  WITH  (NEAR  PAST,  CURRENT,  NEAR  FUTURE) 


WE  WOULD  LIKE  TO  CONTACT  PERSONS  FURTHER  ABOUT  OWL 
ISSUES.  IF  YOU  ARE  WILLING  TO  BE  CONTACTED  VIA  PHONE, 
PLEASE  FILL-IN  THE  INFORMATION  REQUESTED  BELOW. 

NAME: _  PHONE  #  : _ 
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Appendix  B 


DATE:  9FEB87 


PROGRAM  MANAGER'S  OPERATOR  WORKLOAD  ASSESSMENT  PAMPHLET 


OUTLINE 

USER  PROFILE:  The  intended  user  for  the  pamphlet  is  the  program  manager  who  is 
involved  in  both  delineating  the  needs,  and  developing  the  requirements  for  a  new  system. 
This  user  is  not  interested  in  the  details  of  workload  estimation  or  evaluation.  This  user 
also  is  not  interested  in  which  measures  or  which  techniques  offer  the  best  OWL 
assessment.  However,  what  IS  of  interest  to  the  TRADOC  or  AMC  system  manager  is 
high-level  guidance  on  what  are  the  Army  requirements  regarding  workload,  and  what 
high-level  provisions  should  be  built  into  the  system  acquisition  strategy  for  the  assessment 
of  OWL. 

FORMAT:  This  Pamphlet  will  be  structured  to  provide  a  concise,  easily  understood 
presentation  of  the  role  of  OWL  control  in  the  materiel  acquisition  process  (MAP).  Tables, 
charts,  flow  diagrams,  and  specific  examples  will  be  used  liberally  to  promote  quick 
apprehension  of  concepts. 

GOAL:  Provide  the  reader  with  an  overview  of  the  role  of  OWL  control  in  the  materiel 
acquisition  process,  including  the  nature  of  the  problem,  DoD  documents  and  requirements 
concerning  OWL  control,  and  available  technologies  to  assist  the  Army  program  manager 
in  effecting  OWL  control.  Provide  guidance  in  accessing  other  OWL  control  resources, 
especially  the  OWL  Prediction  and  Evaluation  Handbooks. 

LENGTH:  approximately  40-50  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  Definition  of  Operator  Workload  (OWL) 

B.  Requirements,  specifications,  standards,  and  regulations  for  OWL 

C.  Impact  of  OWL  on  Army  Mission  Functions 

D.  Relationship  of  OWL  to  MANPRINT 

E.  TRADOC  Contributions  in  responding  to  OWL  concerns 

F.  AMC  Contributions  in  responding  to  OWL  concerns 

G.  Description  of  the  contents  of  this  pamphlet,  how  to  use  this  pamphlet 

STRATEGY:  Introduce  the  user  to  the  key  OWL  concepts  and  regulations.  Provide  the 
proper  framework  on  how  to  use  this  handbook. 


II.  OVERVIEW  OF  OWL  FUNDAMENTALS 

A.  OWL  Performance  Model 

B .  OWL  with  various  types  of  actions/behaviors 

1 .  Searching  for  and  receiving  information 

2.  Identifying  objects,  actions,  events 

3.  Problem  solving 

4.  Decision  making 

C.  OWL  considerations  for  system  types 
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1 .  Aviation 


2.  C3I 

3.  Air  defense 

4.  Armored/Mechanized  operations 

5.  Maintenance 

6.  Supply 

D.  OWL  considerations  during  the  Materiel  Acquisition  Process  (MAP) 

E .  OWL  Assessment  Program 

1 .  Prediction  (analytic  approach) 

2.  Evaluation  (empirical  approach) 

3.  Analysis  of  results 

F.  OWL  Control  Plan 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  key  areas  and  steps 
involved  in  OWL  prediction/evaluation  and  its  relationship  to  the  materiel  acquisition 
process. 


III.  OWL  IN  REQUIREMENTS  ANALYSIS/CONCEPT  FORMULATION 

A.  TRADOC  perspective 

B .  AMC  perspective 

C.  OWL  trade  offs  in  concept  formulation 

D.  Development  of  a  preliminary  OWL  Control  Plan 

E.  Key  OWL  resources 

1.  Documents 

2.  Organizations  (e.g,  HEL,  ARI) 
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3.  Individuals  (i.e.,  HFE  specialists) 

F  The  TRADOC  System  Manager's  (TSM)  OWL  Concept  Formulation  Check  List 

1 .  What  should  the  TSM  be  ensuring  is  accomplished. 

G.  The  AMC  Program  Manager's  (PM)  OWL  Concept  Formulation  Check  List 
1 .  What  should  the  PM  be  ensuring  is  accomplished. 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  requirements  analysis  and  concept  formulation.  Provide  the  user 
the  know-how  to  inte  grate  the  OWL  Control  Plan  with  existing  requirements,  programs, 
and  other  control  plans  that  are  part  of  the  materiel  acquisition  process. 


IV.  OWL  IN  SYSTEM  DEVELOPMENT 

A.  The  OWL  Control  Plan 

B.  Methods  for  assessment  _ 

1 .  OWL  Prediction  (analytic  approach) 

2.  OWL  Evaluation  (empirical  approach) 

C.  Key  OWL  resources 

1.  Documents 

2.  Organizations  (e.g.,  HEL,  ARI) 

3.  Individuals  (i.e.,  HFE  specialists) 

D.  TSM  OWL  Check  List  for  OWL  Prediction 

E.  TSM  OWL  Check  List  for  OWL  Evaluation 

F.  AMC  PM  OWL  Check  List  for  OWL  Prediction 

G .  AMC  PM  OWL  Check  List  for  OWL  Evaluation 
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STRATEGY:  Provide  the  user  a  step-by-step  approach  to  developing  and  managing  an 
OWL  Control  Plan  during  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  Assessment  Program  with  existing  requirements,  programs,  and  other 
control  plans  that  are  part  of  the  materiel  acquisition  process. 


V.  ITERATIVE  NATURE  OF  OWL  ASSESSMENT 

A.  Materiel  acquisition  process 

B.  System  design  decisions 

C.  Evolution  of  OWL  considerations 

STRATEGY:  Establish  the  concept  that  the  OWL  Assessment  Program  and  its  management 
and  control  are  evolving  processes  which  are  modified  as  the  materiel  acquisition  process 
progresses. 


VI.  EXAMPLE 

A.  An  example  will  be  provided  that  delineates  both  TSM  and  AMC  PM  development 
and  implementation  of  their  respective  OWL  Control  Plans. 

STRATEGY:  Provide  the  user  with  a  concrete  example  of  an  OWL  Control  Plan. 


VII.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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Appendix  C 


DATE:  9FEB87 


WORKLOAD  PREDICTION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Prediction  Handbook  is  intended  for  the  system  designer 
during  the  concept  and  early  design  phases  of  the  military  system  acquisition  cycle.  This 
user  is  interested  in  the  different  OWL  measures  and  techniques  applicable  during  early 
design.  This  user  is  typically  the  person  who  (1)  makes  the  decision  of  which  OWL 
assessment  tools  to  use,  and  (2)  adapts  those  tools  to  fit  the  specific  needs  and 
characteristics  for  the  system  of  interest.  To  perform  these  functions,  the  system  designer 
will  identify  the  system  requirements  and  specific  design  objectives  for  which  the  workload 
assessment  methodology  is  needed  and  will  use  "the  matching  model"  procedure  to  select 
an  optimal  OWL  Assessment  Battery.  The  handbook  provides  guidance  on  the 
implementation  of  the  OWL  Assessment  Battery. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
user  in  following  the  methodology  contained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an  OWL 
assessment  program  during  early  system  development.  Provide  the  user  the  know-how  to 
integrate  the  OWL  assessment  program  with  existing  requirements,  programs,  and 
methodologies  that  are  part  of  the  materiel  acquisition  process  (MAP). 


LENGTH:  approximately  75-100  pages 


CONTENTS 


I.  INTRODUCTION 

A.  What  is  Operator  Workload  (OWL)? 

B.  Requirements,  specifications,  standards,  and  regulations  for  OWL 

C.  System  performance  and  operator  workload  (system  requirements) 

D.  Operator  workload  performance  model:  variables/factors  to  consider 

E.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  operator 
workload  assessment  program  during  concept  and  preliminary  design  phases 

F.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY:  Introduce  the  user  to  key  OWL  concepts  and  requirements.  Orient  the  user  to 
the  critical  concept  -  the  OWL  Performance  Model  -  that  underlies  the  methodology  for 
workload  prediction.  Provide  the  framework  on  how  to  use  the  Handbook. 


II.  OVERVIEW  OF  OPERATOR  WORKLOAD  PREDICTION 

A.  What  is  OWL  prediction? 

B .  The  relationship  between  OWL  prediction  &  OWL  evaluation 

C.  Factors  to  consider  for  OWL  prediction 

1. System  requirements,  (i.e.,  system  performance) 

2.0perator  capabiUties/skills/behaviors  required 
3. Stage  in  materiel  acquisition  process 
4.  OWL  performance  model 
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5. OWL  assessment  techniques 


D.  Matching  model  for  establishing  an  OWL  Assessment  Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
prediction  (analytic  approach)  and  its  relationship  to  OWL  evaluation  (empirical  approach) 


III.  OPERATOR  WORKLOAD  PREDICTION  METHODOLOGY 

A.  Determine  system  performance  requirements 

1 .  Justification 

a.  JMSNS  (Justification  of  Major  Systems  New  Starts) 

b.  MIL-H-46855B 

2.  Requirement  source  documents  (cf.,  DoD  Directive  5000.2,  AR  15-14, 
AR  70-1,  AR  70-10,  AR  71-3) 

B .  Determine  operator  actions/behaviors  for  system  usage 

1.  Sources 

a.  Requirement  documents 

b.  Task  analyses 

c.  Expert  opinions 

d.  Comparative  systems 

e.  Mock-ups 

f.  Simulators 

2,  Classification  of  operator  behaviors/actions  required  for  system 
usage(cf..  Universal  Operator  Behavior  Dimensions  -  Berliner,  AngeU, 
&  Shearer,  1964) 

-  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 
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c.  Problem  solving 

d.  Decisionmaking 


C.  Determine  stage  in  materiel  acquisition  process 

1 .  Mission  area  analysis  and  JMSNS 

2.  Concept  and  exploration  phase 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
(decision  making  process) 

E.  Determine  OWL  assessment  program 

1.  Decision  rules  for  selecting  an  OWL  battery 

2.  Decision  tables  for  workload  methodologies 

F.  Analytic  assessment  methods 

1 .  Purpose:  Prediction  of  OWL  &  "chokepoints" 

2.  Sensitivity 

3 .  Representation  of  OWL  issues/behaviors 

4.  Techniques 

a.  Expert  opinions 

b.  Comparisons 

c.  Simulations/Models 

d.  Math  models 

e.  Task  analytic  methods 

5 .  Interpretation  of  as  sessment  results 

STRATEGY :  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an 
OWL  Assessment  Program.  Provide  the  user  the  know-how  to  integrate  the  OWL 
Assessment  Program  with  existing  requirements,  programs,  and  methodologies  that  are 
part  of  the  materiel  acquisition  process. 


IV.  ITERATIVE  NATURE  OF  OWL  PREDICTION 


A.  Materiel  acquisition  process 

B .  S  ystem  design  decisions 

C.  etc.. 

STRATEGY:  Establish  the  concept  that  the  OWL  assessment  program  (developed  from 
using  this  Handbook)  is  an  evolving  program  which  will  be  modified  as  the  materiel 
acquisition  process  progresses. 


V.  EXAMPLES 

A.  Examples  will  be  provided  for  each  of  the  assessment  techniques. 

STRATEGY:  Provide  the  user  with  concrete  examples  of  OWL  assessment  programs. 
Provide  reality  to  the  OWL  prediction  methodology  described  in  the  Handbook. 


VI.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 
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Appendix  D 


DATE:  9FEB87 


WORKLOAD  EVALUATION  HANDBOOK 


OUTLINE 

USER  PROFILE:  The  Workload  Evaluation  Handbook  is  intended  primarily  for  the 
TRADOC  community  as  well  as  system  designer  involved  with  early  developmental  testing 
(DT&E).  These  users  (e.g.,  TRADOC)  are  interested  in  interpreting  the  results  of  the 
various  workload  assessments  conducted  throughout  the  development  cycle  but  are 
primarily  concerned  with  system  evaluation.  These  evaluators  are  responsible  for  test  and 
evaluation  (T&E)  but  are  also  (like  the  designer)  interested  in  how  to  perform  OWL 
analysis.  In  addition,  they  are  more  concerned  than  the  designer  in  the  actual  constraints 
placed  by  real-world  implementation  of  a  workload  assessment.  Such  constraints  involve 
traditional  (and  non-traditional)  limits  on  testbed  resources  (e.g.,  subjects,  time,  and 
funding).  The  T&E  users  are  also  concerned  with  how  to  transform  OWL  data  and 
information  into  recommendations  for  system  design. 

FORMAT:  This  Handbook  will  be  organized  so  to  maximize  its  utility  as  a  working 
document.  Descriptions  will  be  concise  and  to-the-point.  For  those  users  wanting 
additional  background  and  supporting  information,  references  and  appendices  will  be 
provided.  Decision  trees,  tables,  and  charts  will  be  used  whenever  possible  to  assist  the 
reader  in  following  the  methodology  eontained  in  the  Handbook. 

GOAL:  Provide  the  user  a  step-by-step  approach  to  developing  and  implementing  an  OWL 
assessment  program.  Provide  the  user  the  know-how  to  integrate  the  OWL  assessment 
program  with  existing  requirements,  programs,  and  methodologies  that  are  part  of,  the 
military  system  acquisition  cycle. 
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LENGTH:  approximately  125-150  pages 
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CONTENTS 


1.  INTRODUCTION 

A.  What  is  Operator  Workload  (OWL)? 

B .  Requirements,  specifications,  standards,  and  regulations  for  OWL 

C.  System  performance  and  operator  workload  (system  requirements) 

D.  Operator  workload  performance  model:  variables/factors  to  consider 

E.  Purpose  for  handbook:  methodology  for  determining  and  implementing  an  OWL 
assessment  program  during  the  design  andevaluation  phases  of  the  materiel 
acquisition  process. 

F.  How  to  use  this  handbook;  overview  of  contents. 

STRATEGY :  Introduce  the  user  to  the  key  OWL  concepts  and  requirements.  Orient  the 
user  to  the  critical  concept  -  OWL  Performance  Model  -  that  underlies  OWL  evaluation. 
Provide  the  proper  framework  on  how  to  use  the  handbook. 
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II.  OVERVIEW  OF  OPERATOR  WORKLOAD  EVALUATION 


A.  What  is  OWL  evaluation? 

B .  The  relationship  between  OWL  evaluation  &  OWL  prediction 

C.  Factors  to  consider  for  OWL  evaluation 

1.  System  requirements ,  i.e.,  system  performance 

2.  OWL  assessment  program  constraints 

a.  Subjects 

b.  Time 

c.  Funding 

d.  Etc. 

3 .  Operator  capabilities/skills/behaviors  required 

4.  Materiel  acquisition  process 

5 .  Earlier  OWL  assessments  (OWL  prediction) 

6 .  OWL  performance  model 

7 .  OWL  assessment  techniques 

D.  Matching  model  for  establishing  an  OWL  Assessment  Program 

STRATEGY:  Provide  a  global  "mental  map"  for  the  user  on  the  steps  involved  in  workload 
evaluation  (empirical  approach)  and  its  relationship  to  OWL  prediction  (analytic  approach), 
materiel  acquisition  process,  and  system  design. 


III.  OWL  ASSESSMENT  (TEST  AND  EVALUATION) 


A.  Determine  system  performance  requirements 

1.  Requirement  source  documents(cf.,  DoD  Directive  5000.2,  AR  15-14, 
AR  70-1,  AR  70-10,  AR  71-3 

B.  Determine  operator  actions/behaviors  for  system  usage 
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1 .  Sources 


a.  Requirement  documents 

b.  Task  analyses 

c.  Expert  opinions 

d.  Comparative  systems 

e.  Mock-ups 

f.  Simulators 

2.  Classification  of  operator  behaviors/actions  required  for  system  usage 
(cf.,  Universal  Operator  Behavior  Dimensions  -  Berliner,  Angell,  & 
Shearer,  1964)  -  Chart  of  Universal  Operator  Behaviors  - 

a.  Searching  for  and  receiving  information 

b.  Identifying  objects,  actions,  events 

c.  Problem  solving 

d.  Decision  making 

C.  Determine  stage  in  materiel  acquisition  process 

D.  Determine  OWL  performance  model  factors  to  incorporate  into  matching  model 
(decision  making  process) 

1 .  For  example,  environmental  factors  such  as  noise,  vibration,  heat,  and 
cold 

2.  Sustaining  period 

E.  Determine  OWL  assessment  program 

1 .  Decision  rules  for  selecting  an  OWL  battery 

2.  Decision  tables  for  workload  methodologies 

F.  Establish  framework  for  the  evaluation 
1,  Task  scenarios 

G.  Empirical  Assessment  Methods 

1.  Purpose:  Evaluate  design  decisions 
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2.  Sensitivity 

3.  Task  scenarios:  representation 

4.  Techniques 


a.  Operator  opinion 

b.  Primary  task 

c.  Secondary  task 

d.  Physiological  responses 

5.  Interpretation  of  assessment  results 

STRATEGY:  Provide  the  user  a  step-by-step  approach  to  implementing  an  OWL 
assessment  program  -  test  and  evaluation.  Provide  the  user  the  know-how  to  integrate  the 
OWL  assessment  program  with  existing  requirements,  programs,  and  methodologies  that 
are  part  of  the  materiel  acquisition  process. 


IV.  ITERATIVE  NATURE  OF  OWL  EVALUATION 

A.  Materiel  acquisition  process 

B.  System  design  decisions 

STRATEGY:  Establish  the  concept  that  OWL  assessment  (test  and  evaluation)  is  an 
iterative  process  that  is  conducted  throughout  the  materiel  acquisition  process  to  ensure 
OWL  is  not  a  system  problem. 


V.  REFERENCES  &  SUPPLEMENTAL  MATERIALS 

STRATEGY:  Provide  relevant  references  and  source  material  that  offer  the  interested  reader 
additional  information  on  OWL  issues.  Include  annotations  and  advice  as  appropriate  to 
guide  users  in  access  to  and  interpretation  of  materials. 


APPENDIX  E 


OWL  INFORMATION  SYSTEM 

An  OWL  Infonnation  System  is  under  development  to  provide  for  the  control  and 
analysis  of  the  mass  of  documents  and  other  resources  comprising  the  OWL  scientific 
database.  Currently  under  implementation,  the  system  is  comprised  of  several  components. 
One  of  these  is  a  computerized  database,  to  collect,  to  organize,  to  analyze,  and  to  retrieve 
relevant  citations  and  information  associated  with  those  citations.  For  convenience,  we  will 
discuss  this  aspect  in  terms  of  the  OWL  Information  Data  Base  and  the  OWL  Analysis 
System.  The  second  part  consists  of  a  library  consisting  of  the  physical  documents  keyed 
to  the  database  for  easy  retrieval. 

Overall,  this  effort  constitutes  part  of  our  effort  for  the  development  a  set  of  useful 
tools  for  the  analysis  of  operator  workload,  providing  decision  aids,  and  an  information 
support  system  for  practitioners. 


OWL  INFORMATION  DATA  BASE 

The  data  base  contains  standard  bibliographic  information  on  each  report  or  article. 
The  software  chosen  is  dBASE  in  which  provides  a  convenient  and  widely  used  relational 
database  and  program  system  for  IBM-PC's  and  compatible  machines.  The  hardware 
environment  was  selected  to  be  broadly  compatible  with  government  standard 
microcomputer  systems.  The  overall  system  is  composed  of  several  data  files  and  a  number 
of  accompanying  dBASE  III  programs  to  access  and  manipulate  the  data  files. 
Development  of  the  system  to  this  point  has  been  done  carefully  to  provide  flexibility  in 
retrieving  citations  based  on  particular  user  needs.  Some  additional  programs  will  be 
useful,  but  these  are  being  done  as  the  need  arises. 

The  OWL  reference  database  now  numbers  about  1400  reference  items  and  is 
growing  steadily.  There  are  several  additional  bibliographies  which  have  not  yet  been 
entered,  for  example,  one  created  by  Douglas  Aircraft  Company  which  consists  of 
approximately  700  items  and  is  supposed  to  be  available  soon  in  dBASE  III  format.  As  we 


review  the  actual  documents  in  the  database,  additional  references  of  interest  are  being 
collected  from  the  reviewed  documents  as  well  as  from  direct  contact  with  authors  of 
documents  already  entered. 

The  database  system  was  designed  to  access  the  information  in  a  number  of 
different  ways  to  accommodate  various  users,  those  working  on  the  project  and  more 
importantly,  those  who  will  be  in  need  of  the  information  when  the  project  is  completed. 
Each  bibliographic  record  is  composed  of  a  number  of  fields  as  shown  in  Table  E-1.  A 
form  has  been  prepared  to  facilitate  preparation  of  material  for  entry  into  the  database 
(Figure  E-1).  Of  note,  are  the  keyterm  (in  title)  fields  which  facilitate  access  by  title. 

A  large  number  of  reports  are  possible  to  generate  citation  listings.  The  report 
programs  which  have  been  developed  to  facilitate  access  of  the  database  are: 

•  Complete  report,  alphabetized  by  authors.  This  lists  all  citations. 

•  Report  for  all  citations  for  a  single  author 

•  Report  selecting  by  keywords,  such  as  "physiological  measures" 
whereby  a  report  is  generated  that  lists  all  citations  under  this  topic. 

•  Report  listing  of  references  with  copies  in  house 

•  Several  utility  reports  for  editing,  etc. 

These  report  programs  currently  provide  bibliographies  in  two  standard  forms:  Human 
Factors  Society  and  American  Psychological  Association.  The  programs  are  developed  so 
that  others  could  be  added  easily.  Additionally,  several  programs  have  been  developed  to 
enter  and  edit  the  reference  file  along  with  necessary  utility  programs.  The  development  of 
these  programs  allows  the  user  to  enter  additional  references  and  reviews  and  still  maintain 
the  system  easily. 


OWL  INFORMATION  ANALYSIS  SYSTEM 

While  the  bibliographic  file  will  be  of  general  use,  the  more  important  part  of  the 
data  base  consists  of  a  separate  file  containing  the  reviews  produced  by  members  of  the 
Analytics,  Inc,  OWL  team.  The  reviews  are  based  on  objective  and  standard  classification 
techniques.  The  review  form  is  shown  in  Figure  E-2  and  the  structure  of  the  review  data 
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file  is  shown  in  Table  E-2.  This  analysis  follows  the  taxonomy  presented  by  Wierwille  and 
Willi ges  (1978)  which  was  in  turn,  modeled  on  the  taxonomy  developed  earlier  by 
Berliner,  Angell,  and  Shearer  (1964).  We  have  augmented  this  analysis  with  the  addition 
of  statistical  and  observer/subject  categories,  some  of  which  were  suggested  from  an 
analysis  done  by  Douglas  Aircraft  Company.  Ultimately,  most  papers  will  be  reviewed. 

The  reviews  are  entered  into  a  second,  separate  data  file,  linked  to  the  reference  file 
by  item  number  (and  first  author  as  a  check).  (Item  No.  is  an  arbitrary  number  assigned 
principally  on  order  of  entry.)  The  use  of  separate  files  permits  multiple  reviews  (several 
reviewers)  on  a  given  paper  and  takes  advantage  of  the  relational  database  properties  of 
dBASE  ni.  To  date,  reviews  for  approximately  100  papers  have  been  entered;  this  aspect 
will  receive  considerable  attention  in  the  near  future  as  we  enter  into  Task  3. 

As  the  number  of  reviews  entered  into  the  system  increases,  we  will  develop  a 
report  generator  to  permit  the  user  easy  access  using  the  information  contained  in  the 
reviews. 


OWL  LIBRARY 

The  OWL  Library  is  being  assembled  through  both  formal  and  informal  sources. 
The  sources  include  publications  available  through  DTIC  and  NTIS,  journal  articles, 
conference  proceedings,  conference  papers,  monographs,  dissertations,  etc. 

Additionally,  documents  are  being  sought  directly  from  well  established  workload 
laboratories,  e.g.,  NASA  Ames  and  NASA  Langley,  Air  Force  AMHRL,  etc.  Also,  letters 
are  being  sent  directly  to  authors  already  in  the  database  to  solicit  copies  of  their  work  as 
well  as  any  work  which  is  new  or  was  inadvertently  missed  on  the  first  pass  through  other 
sources.  Currently  we  are  receiving  20  to  30  documents  a  week  from  authors  with 
approximately  20%  being  additions  to  the  reference  database. 
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TABLE  E-1 


Listing  and  description  of  fields  in  OWL  reference  database 


Description _ FIELD  NAME  TYPE  LENGTH  DECIMAL 


Owl  database  # 

riEMNO 

N 

8 

Review  done 

REVIEWED 

C 

1 

Lib.  Catelog  # 

CATALOG 

C 

12 

Report  No. 

REPORTNO 

c 

25 

Title  -  Article 

ARTICLE 

c 

250 

Title  -  Book 

BOOKTITLE 

c 

250 

First  Author 

AUTHORl 

c 

25 

Second  Author 

AUTHOR2 

c 

25 

Third  Author 

AUTHORS 

c 

25 

Fourth  Author 

AUTHOR4 

c 

25 

Fifth  Author 

AUTHORS 

c 

25 

Corp.  Author 

CORPAUTHOR 

c 

100 

Vol.  Editor 

EDITOR 

c 

60 

Book  Co. 

PUBLISHER 

c 

80 

Journal 

JOURNAL 

c 

80 

Year  of  Pub. 

PUB YEAR 

c 

4 

Volume 

VOLUME 

c 

8 

Pages 

PAGES 

c 

13 

Keyterm 

KEYTERMl 

c 

50 

Keyterm 

KEYTERM2 

c 

50 

Keyterm 

KEYTERM3 

c 

50 

Keyterm 

KEYTERM4 

c 

50 

Keyterm 

KEYTERM5 

c 

50 

Do  we  have? 

INHOUSE 

c 

1 

Notes 

NOTES 

c 

250 

Date  entered 

DATE 

c 

E-4 

8 

3 


HGUREE-l 


OWL  REFERENCE  ENTRY  FORM 


Item  No. _ (Assigned  by  computer) 


Author  1 :  _  Author  2 : 

Author  3  :  _  Author  4 : 

Author  5  :  _ 


CORP.  AUTHOR: _ 

(Use  if  no  authors) _ 

TITLE:(Joumal,  Article,  Book,  or  Chapter) 


EDITOR(s): _ 

BOOKTITLE(For  editted  books,proceed.): 
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REPORT  NO. _ 

JOURNAL: _ 

PUBLISHER:(Use  only  if  Booktitle  used;  Place  &  pub.  company) 

Year  of  Publication: _  Volume  (&  No.): _ ( _ )  Pages: 

Keyterm  1  : _ 

Keyterm  2 : _ 

Keyterm  3  : _ 

Keyterm  4 : _ 

Keyterm  5 : _ 

Copy  of  Paper  in  house?  _  (Y  or  N) 

Notes:  (250  characters) 
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Table  E-2. 


Description 

FIELDNAME 

TYPE 

LEN 

DECIMAL 

GENERAL 
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Reliability 
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Due  to  the  limitation  of  a  maximum  of  128  fields  in  dbase  III,  a  number  of  the 
characteristics  had  to  be  combined  into  a  single  field.  The  rules  for  this  are  based  on  the 
estimated  probability  of  searching  within  a  category.  For  example,  Math  models  from  the 
analytic  taxonomy  is  a  single  field.  The  subcategories  under  math  models  are  a  combined 
field.  If  Math  Models  is  entered  (something  other  than  blank)  then  you  would  be  asked  for 
the  subcategories  and  these  would  be  stored  in  TAXA41,  in  order.  Thus,  if  Infomation 
Theoretic  were  the  only  subcategory,  the  contents  of  TAXA41  would  look  like  this  '.X.'. 
(Periods  are  put  into  the  field  when  there  is  no  entry.  This  is  done  so  that  if  the  whole  field 
is  printed  it  is  easy  to  tell  where  the  entry  is.) 


In  the  main  table,  combined  entries  are  indented.  Those  fields  which  are  combined 
into  another  label  are  blank  for  field  type  and  length.  The  example  below  illustrates  this. 
TAXA42  and  TAXA43  are  combined  into  TAXA41. 


Math  Models  TAXA4 


C  1 


Manual  TAXA41  C  3 

Info  Theor.  TAXA42 
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1 


Other  TAXA43 

Further,  as  the  indentation  above  represents,  an  indented  field  will  be  empty 
automatically  if  the  main  field  is  blank.  Thus,  if  TAXA4  is  blank,  the  subcategories  will 
also  be  empty  e.g.,  TAXA41  would  be 

In  a  case  where  there  are  subsubcategories,  these  are  listed  in  the  the  logical  order 
following  the  subcategory  driving  it.  TAXE31  has  a  subcategory  of  TAXE311,  ...,. 
TAXE311  is  a  field  containing  these  entries  and  follows  TAXE31  on  the  dbase  III  file 
listing.  TAXE32,  etc.  is  stored  in  field  TAXE31. 
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This  form  will  be  used  until  our  online: 
_ database  system  is  available: _ 

Instructions  to  Reviewer: 


If  Item  No.  below  is  blank,  please  provide  full  reference,  including  all  authors, 
title,  journal  or  volume,  pages,  publisher,  year,  etc.  so  we  can  enter  the  reference  into  our 
database,  along  with  your  review. 

We  have  endeavored  to  design  this  form  to  be  as  easy  to  use  as  possible.  In  the 
following,  please  check  those  items  which  are  relevant  to  the  paper  being  reviewed. 
MORE  than  one  item  in  a  category  may  be  appropriate,  so  please  check  all  appropriate 
items. 


Try  to  be  as  complete  as  possible.  Remember,  others  will  be  using  the  information 
you  submit. 

For  articles  which  are  difficult  to  locate,  it  would  be  appreciated  if  you  would 
provide  a  copy  for  our  files. 
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